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FROM 



NIH Grants Peer Review Study Team 




Submission of Study Team Report 



The'NXfl Grants Peer Jleview Study T^am Ms pleased to submit to you 
the attached report. This report / Phase I, is based on a careful-^ 
study ot the issues by the Study Team and others at NIH and, in 
part, on the testimony presented at three public hearings, the 
letters received from the scientific community and the general 
public, and\the' results of a survey of NIH review. group members,^ ^ 
Phase II, to be completed in late 1977, will present, a detailed 
analysis and evaluation of the letter responses and testimony 
from the public hearings; and will include 'the completed analysis 
and evaluation of the surT*ey of IRGs and Council members, 

The Study Team's^^port and recommendations are presented in 
Volume I. Volume II presents background appendices and Volume III 
is a collection of supplementary reports. *^ 

We have prepared a series of recommend4.tions which we believe are 
central to the goal of not only maintaining but also improving the 
quality of the peer review system and thus of the biomedical research 
supported by NIH. We wish to call particular attention to several 
recommendations which represent er marked departure from the current 
peer review procedures or are deserving of special consideration : 

o That a formalized KlH Grants Peer Review Appeals 
System be established, central to which is the 
establishment of a j>osition of OMBUDSMAN to l>e 
appointed by the Director, NIH. 

o.. That NIH' should periodically announce- all upcom- 
. * ing vacanciea on initial review gr1)ups and inylTte 

suggestions regarding candidates for specific groups* 

o That authority to establi^sh or discontinue Initial 
' ' Review Groups should be delegated to the Director, NTH. 

o That NIK should seek to have the authority for ^ I 

• selection and appointment of mfeinbers of Advisory 
Couxicils/Boards delegated to the Assistant 
Secretary for Health, HEW. 



Director, NIH ^ ■ " 2 

o That those portions .of the meetings of advisory ' 
•groups ^ich involve the review' of grant applica- 
tions shguld continue to be closed tf)« the p^blia 

* (including those submitting applications) ^ either 
under cur;rent exemptions to the open toeeting 
requirement or t^lrojffgh legislation. 

0 That, as soon as practical after a National Myisory 
douncil or Board completes the review qf a grmnt 
application, the Bureau, Institute, or Division 
Should tqutlnely send the associate4 summary state-- 
ment with the priority, score displayed to the 
principal investigator ^ named in the application. 

o That the Director, NIH, should take immediate steps 
to limit the workload of all Initial Grant Review 
Groups to a level Compatible with maintaining the 
high quality of review. 

The^tudy Team als'o^addressed and developed recoimnendations concern- 
ing d variety of other key issues. These Include identification and 
special consideratloti of ^unorthodox rq^earch approaches, the need 
for a single priority "score convention for use throughout NIH in' 
'lieu o£ the present dual score system, conflict o£ interest procedures 
applicable to^ review group members, membership of employees of profit- 
making organisations in initial review groups, extensioti of peer 
review procedures to cover the assessment of business management 
^practices ip large grant application^, and opportunities for improving 
the "grants peer review system througih a continuing program of prospec- 
tive studies involving specific 4>rocedures. Furthermore, the Study 
Team believes that a^l of the issues it considered are important and 
that, taken together, its recomm^dations form a self-Consistent ' 
ensemble of actions which will donmcl» to insure that bhe NIH grants 
peer review system remains a model of scientifijC and administrative 
excellence^ 

We believe that the report ^of the Study Team, and particularly its 
recommendations, will be' of considerable' interest to the biomedical ' • 
comsminity, the gen^ral public, and the NIH extramural staff. We have 
copies available fox' distribution if you wish to share the report with 
your staff or ^visors. 



Director 



Menber^ of thf Study Team are ready to assist you and your dtaff In 
the Implementation of the reconmendatlons presented Iti this report • 
If you so wish. >^ . ^ , 




Ruth \f. urschsteln 
Chairperson 



Robert P. Akers 




Carl A. Fr^tts 



Norman D. 06ry 




William H. Goldwater 





Mathllde^ Solowey 
Executive Secretary , 



Ann A.llCaufman 




William F. Raub p 
/George F. Rusaell, Jr. ^ 
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PREFACE 



The IJIH Grants Peer Review Study Team is pleased t,o present this r^rt 
to the Director, NIH, for hls-€onsic^eration. The April 28, 1975, statein^it 
of the Acting Director, .Nlflf established and set forth the charge to the 
Study Team.^ Ift-CHie of a series rof assigned responsibilities, the Study 
Team was asked to "Examine In critical detail the entire process ot peer^ 
review and make, whejre necessary, reconmeidations for modif icatibns or 
diange...."^ It is hoped that the present study will be one of a nuraber 

other studies of the NIH peer review systen, and tihat other study 
groups will address aspects pf the peer review process not covered in 
this- report. 
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Tte NIH grants pfeer review system is- and has been extrenely 
effective in identifying bianedical r^esearch activities of 
high quality 
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•Sbe principle of s^ration of grant application review 

and program staff functions at NIH is- strongly endorsed 4o 
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NIH should develop detailed instructiwis for determination of ' 
conflict of interagt for menbers of review and advise^ groupfe 100 

KIH should adopt a procedure under which Form IffiW 474 . 

(Confidential Statement of Enploypsait apd Financial Interest?) is • - ' 
' returned to a'jnember, as inconplete, when such iqeniber makes no, 

mty in a section of the form but does not write "ndie" or some 

/equivalent.. ; 101 

• ' ' > 

All executive secretaries of IBGs should be givai access to and 
.should review, at least annually^ the Form HEW 474 of raentiers and 
• should -be required periodically to kttend traini^ sessions on ' ' 

evaluation of conflict of intetest situations...-..". ^ "101 

« ' ^ ' ^ * 

All initial invitations to serve on review and advisory groups / • 
should make it explicit that final a^jpointanent is contingent upon 
revie*/ of the' completed- Form HEW 47^ for conflicts of interest, or 
potential coftflicts of interest, and that the newf member should 
formally notified of the appointment after "the appointmait process 
(including review of the Form HEW 474 J has beeR ccnpreted 10^ - 

Hi* names of new rwnbers of review and advisory gfbups 'should not ' ^ 
be released to' the news media or other members of/the public until * 
'such time as final appointment has occurred aftft^ completion of 
review of the HEW 474. ..v.V...w>«. ........... 102 



^^PPO^^toait of EmployeelB of "Foc^rofit' Organizaticais to NIH Initial* 
^ Revlenf Groups., - 
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Btployees of '^fdr-prpfit'.organization^ should' be eligible f<i 
membership on all initial review groups considering grant ' 
applications (including National Research Servifc^ Award 
OTlications).,ii3he, basis for selection of such scientists shall be 
-•the saibe as for those employed by "non^rQfit",'organi?atioite......... 105 

■ •* ' ' 

All "oanflict of interest" statements (HEW Form 474)[, au^itted by 
enployees of profit-making organizations-mdst be reviewed by the" 
agency head, v*io must be satisfied that the individual can .serve 
on the specific committee to which thajt pefson h/us been nominated" * 
without being in violation of the conflict of 'interest statues., ^105 
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Legal Cong ider at ions Regarding Grauits Peer Review . 
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Ihe Public Health Service (f«S) Act should be amended to provide 
statutory, exeooption frcw the^requireihents of the Freedom of 
. Information Act (EOIA) for disclosure of research designs euid . 
protocol^ presented . in grant applications'. 116 ' 

Ihose portions of the meetings of advisory groups whicdi. involve 
the review ,of grant applications should continue to be closed to 
the public (including those submitting applications), either under v , 
currwit >xeiii«:ions to the opbn-meeting requirement or through a 
statutory aroendmeht..... ',...[.. ••• 

The current system of dual review should be preserved, with grant 
f plications being reviewed first by initial review groups consisting 
' soltXy of scientific and tedinical, experts and then by NationSd 

Advisory Councils and Boards including representatives of the * 
, public. ••• V 

In releasing reviewers' opinions under the Privacy Actj. an adequate , 
legal basis should bo estlfclish^d for protecting the reviewers' 
^^pcnyraity either through reinterpoetation of existing law, t5r 

* It of new legislation 124* 



•Jhe NIH and HEW should establisl/ a mecAianisip for special, periodic - \ / 
' assessroent-ef the inpact of this new legislatiwi ("sunshine lajfi^) on / ^ 
the quality of -grant applications and on the quality Of peer review — ^ 
of such applications. Such assessro^t should be r^rted to the 
legislative and executive brandies of the Goverranent r ^ 125 

Inpact of Review Workload' on Quality of Initial Merit Review 127 

The, Director, NIH, should taKe immediate steps to limit the workload. ^ 
of all initial grant review groups to a level which is compatible , . 
' with" maintaining the high quality of peer review.. 132' 

Authority to establish or discontinue initial review groups as the > 
' peer review workload dic±ates should be delegated to the Director, > 
OTH,;.. 132 

Additional resources should be provided for peer review of grant 
jpplicationjs v^ere acceptable alternative approadies to reduction :« 
of worklodB will pot permanently ar^a effectively "resolve long- * 
standing review overj/oads 

The Director, NIH, should establish a permanent roe<±anisin to ^ 
determine an appropriate ceiling or maximum workload fir eac* ^ 
NIH Initial' Grant Review.Group \ 133 
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WH-viae S tandards and Guidelines for Peer 'Review Prdcedures .. 14 

• • The proposed peer review regulations prepared under Section 475 
• <« the. Public Health Service Ac^. by HEW should be fi/ialized... 14 

NIH standards and guidelines' should be prepared or revised >s *_ 
soon as possible after .consideration and evaluation of . 
, reconmendations mhde in the study by the Executive Secretaries 
. Review Activities Oartnittee * . . ; • 24 

In order tojnaintain 'and improve th.e level of excellfence of the ' 
grants peer review system at NIH, a staff position should be 
established within the'Office of Extramural Research and Training 
ts^rovide for quality assurance of the system , 146 

12^"*? curricula should be developed by NIH for extramural program, 1 

aid review^ staff in order to provide orientation and to refresh '\. 
and reiterate principles concerning the philosophy, objectives, 

» \^and pr<)cedures for peer review ; •. 14g 

Open forians and workshops for program and review staf £ i^hoald be 
established on a continuing basis vso as to encourage and inprove 
exchange and communication of ideas concerning issues relevant 
' to t^e^ peer review, system^ , 146 



Orier^iabion ^sions. should^ be held annually for all Initial 
» Review Grotq? and Council, members by the DirecCortUlE, the 

BID Directors and their staffs ^.^ 14g 

Review of. Business Management Practices ^ I55 

It should be recognized as an NIH policty position that the use of 
business. management consultant^ as reviewers is ^ necessary 
adjunct to the scientific review of the large, coraplex, and 
ndltif acted program projects, ^ centers, or of those grants 

- involving consortium arrangements . Considering the nature of the 
traditional, ^investigator-initiated research proj^ application 
and its review, the use of business^management consultants 

. ^ould not usually be considered necessary or e^ropriate -..i 160 

• Ihe principal . criteria used to .determine the need for business 

management consultant support should be. related t§ the odfeniza-: " 

tionpl or administrative cfarplexity existing in any particular 

grant apt>llce\,tion • 260" 

NIH should develqp a policy issuance covetir^^e rble of and Aeed 
for assessm^t of busineSss management in the review of the lirge 
program pr-oject and center grant applications and uniform guide-' 
lines to be ^ollcfwed by the l?usirjess management consultants in 
their roles as roeniiers of project site visit teams and advisory 
' groups : y 162 
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Business management consultants should contribute to overall 
reooBmendations on project site visits and ati Initial Review Group 
meetings, but they phould not vote or give a priority rating. In 
addition, they shoul'd prepare a specif ic -portion o_f prbjecfsite 
visit reports and/or IRG summary statemaits vihich should be 
recognized as separaj:e from "scientific review; and evaluation, but 
should be carefully cwsidered in reac^iing r^iconineidatiOTS for 
•each project. ....!.».•>.. 



Procedures should be developed vrtiidi would a 
business management consult^ts, including 
for management improvement, to be made 
institution '. •, 



Grants management staff of the BIDs should f«irticipate vith the 
IPG executive secretaries in the determinatim of need for and 
selection/bf business manfgfement consultants! for review and when- \ 
ever possible, they should .jfccoipany the sit^ visit teams <5n reviews 
the Gcnplex projects as stiff resoiarces only 



low the reports of the 
sbecifi-c recoitmendatioris 
avail, iblQ to thfe applicant- 



163. 



164 



165 



The Role- of Peer Review in Support of Unorthodox, Innovative Beeearch. 167 



NTH should; 



(1) Require the applicant to identify and support in detail 
any contention that the research project being proposed 
is unorthodox or innovative; 



171 



(2) ' request initial review groups^ to identify applications 
they ccxisider to. be e^)eciaJ.ly unorthpddx or innovative, 
vrtiether or not the applications wer^ sjo identified by 

^ the e^licants; 



(3) encourage IRG members (as a group jorl individually) to 
prepare a statement m addition td> the regular summary 
statement pointing out the unorth/xadx of innovative aspects 
of the applicatiOT and its signii^c^ce. 

(4) consider the feasibility of developing an experiment 

• involving limited support for Certain Speculative, high- 
risk, 4anorthodox or innovative research proposals. Sucii 
a study might be part of a largter, much-needed effort to . 
examine the processes of decision-racking irv allocating • 
researdi support.,..,...,... 
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Belease oC. Summary ^ Statements to Principal Irtvestigators 177 

Suraqiary statements (with the, priority scores displayed)* concerning' 
grant applicaticms should be routinely sent to the principal * 
investigator as soon as practical after conplfetion of the review 
, the particular National Advisory Council/Board. / This ' 
recoranehdation, is contingent upon the understanding that these 
doojments^wlll be released only .to thtse individuals. If this 
procedure could reasonably lead to a requirement that NIH make > 
these dcx:uments available to applicant instituticms then the 
suinwry statements should be released cmly igpon receipt of an . 
appropriate requesfl f rem the individual concerned ..TT.'. 178 

When a sumnary statement is released, upon a principal 

invest igator*s^ requeist,- prior to'final action by a National ... ^ * - 
/ AdvisorJ^ CouncU/Board, the docurtent should be:i>rovided with the 
^ apriority scortr*displayed.and the requestor should be advised * ^ 

♦ that the information is interim- in nature and that any attenpt 
to mqdify the original^pplication or provide comtneritary for 
^ considerations by the JCouncil/Board may result in deferral 6f 

consider at ion of the fippl icat.ion 180 

J r y ^ . 

NIH should request author izain.on, through either, regulation or ^ y 

legislatiai as, 'appropriate, to release an initial review group ^ — ^ 



fsuitmary Statement to the%pr incipa* investigator named in the _ 
application only after the review' of his/her grant application 
( ' is complete, i.e., after Council/Board action. 181 

Pri,or ity Scores on Summary Statements . . ^. 183 

A ""^irqle priority- score" convention should be adopted throughout 
NIH butV until this occurs only the type of priority score used 
^ by the^ID to make decisions should be di?pla^ on the suninary 

statqwepts released to principal investigator^.,...., ' 184 

' " f ^ ,^ ^ ^ . . ' \ 
dtie system of develo|)ing priority scores as a numerical indicaticffi ^ 
of scientific merit of grant applications should be studied in 
^order to assess Aether the present procedure should be retained 
or a new procedure is needed....^ 184 

/ NIH should ccwJuct ^udies of: 

(1) Variations in individual reviewer and review group behavior 'tn 
r ; r^tr^ applicaticma, over tiite, and among different "IRGs, and 

of the factors iwhich act to increase or decrease such variability; ^ 



(2)/ Variations in the quality of gVant applications assigned to a 
pgtven. li^ from one review round to the next, over time; and 
I vaj^iatiohs in the quality of griuit c?3plications assigned to 
the IRGs; " ^ 
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(3^ Ihe effects on the review proces^ of displaying the raw ^ • . 

* _ priority score to the initial review group members 

'iHpediately after they assign their individual scientific 
•mefU ratings, an^ of giving them the option, at that point, ■ 
to rwpen discussion and rerSte the application.... 

- ^ - - - " • 1 fi7 

Grants Peer Review. Appeals System . 

• '■ ~ ' . ' 

A formalized NIH Grants Peer Review Appeals System should be 

' established to Correct or eliminate the deficiencies noted 193 

' ' - / ^ • ° , 

An OtBLDSMAN shou^ be ap^ihted by the Director, NIH.,' i94 

f TO provide the needed higher levels of review related to appeals, 
^ a Granjis Peer Review Appeals Board U permanent comnittee), should . 
Be established - '•• 

Specific criteria should be established for reconsideration (appeal) 

of NIH action on grant 'eqppl icatibns • • • •'•'^ 

Mechanisms and procedures. should be established for appeals 

concerning assignment of grant applicaticMis...,;.-. 196 

• ^ Mechanisms and. procSur^s'^SjiOuld be established for appeals 

concerning scientific reviev>of grant applications.... V^' 
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Hie? pa^t 20 yearsjtiave seen a demonstrably growth of the extramural grants * \ 
program .supported by the National Institutes of Health (NIH)r Specifically, - 
support for NIH research' grant awards grew from $32 million in T£ 1955^ to 
$473*millipn in FY 1965, to $1;^12 billion in FY 1975. In 1975, ll;243 ' , ' 
ccnpeting research grant applications* were reviewed by peer review groups*. ^ 
Additionally, several thousand other kinds of applications w^e ^received, t , 

such.as individual arxJ instituticMial' National Research Servicfe Award . . 

applications, resource applications,. and reseatch career program 
'awards. In 1975, the traditional investigator-initiated ifesearch grant appli-' ' . ' 
cations requested a tot.al of $887 million (for the first year only of ' . 
molti-year applications). As a reSult of peer review evaluations, 
$431 iftil^ioaw^re recommended for approval and $318 million inNfi^st- . 
year ditect-cost funds were awarded for cotpeting projects. ^ 

Uie level of interest, in the NIH extramural res$c\rch prograj|> and *its 

operation is exceedingly high in the biomedical scientific dommuffity, 

whidi-is increasingly dependent for cesearch .support on the availability^ ^ • 

of federal funds. Bqually high interest is shown by the gef^ral' public, 

which looks to this use of federal funds^ for support of biomedical research, 

as a means of ultimately improving the health care of the natipn., . ^ ^ . ^ 

Ihe extramural research program Hs essential to the mission of NIH and ^ 

it is therefore equally essential that it.be of the highest quality. " ' \ 

Since the inception i?f the extramural program at NIH, the gauge for 

selection of research and other grant activities fOr support has been by^ 

rtve^ps of peer reveiw. Traditionally, NIH routinely monitors and .administers^ 

'the peer review system to- assure its continued quality and effectiveness - 

'and to provide a continuous process of improvement. 

Early in 1974, a two-day seminar was .held by NIH officials to examine and 
appraise the extramural research and training grant review system, arid the 
use of peer review and public advisory groups by NIH for its rapidly ' 
growing extramural research grants program. 

A secies of reconmendations emerged fran the seminar, including one 
specifically directed toward the need for an NIH Study Team to continue 
the studv^of the peer review syst^. Responding to the recommendation, 
in the spring of 1975, the Acting Director of NlH established the NIH , " 

Grarits Jeer Review Study Team (GPFST) consisting primarily of NIH personnel 
and incnjding a meittoer of the Office of General Counsel (OGC), DHEW, 
;assigne<S to NIH. The Acting -Director designated for membership pn the 
study Team,- individuals whose Expertise hie considered de^^r^ble and. necessary 
to acconplisti the objectives. - 

y 

Summary of the Charge to the Study Team 

The Study Team was asked to: . ^ 

. • donduct a detailed and comprehensive study of the NIH peer ^ 
• review system. This study, in broad outline, should focus pn the 
philosoE*iyju>d procedures of peer review^ its applicability to the ^ 
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mi awaraing- instruments and programs, the attributes and problems' 
of alternatives,, and the role and charactec of . peer r-eview in the* 
^ decisiort-makipg process at-NIH." ^ ■ . - 

, More specifically, the Study Team 'was asl^to: * ' 

"Ex^^e in critical detail, the -entire process of peer review and ' 
raak«, *)here necessary, recaifnendat ions' for modifications or 'change. 
In addition to other items wifich tte Study Team may wish to consider, 
specific attention should" be givefh^toj . {!) selection procedures and ~ ' 
» • ' criteria for advisory cormittee members; (2) the need and feasibility 
tor an NIH policy dfefinind" the administrative and scientific 
^ relationships of the staffs responsible, for the technical merit 
review of gra^t applicati^jns^conducted in «tfirding Institutes and 
. ^^Vlt^ those responsible for scientific program management; 
(J) taie need, advantages, -and disadvantages for a t^hnical melit 
review "appeal" mechanism for appli^ai^ts ^,an assessment of the • . 
process currently in use;H4) thejrole and character of peer review 
- . in the decision«fnaking process at NIH; (5)" the capability of the peer * 
• review system to accomnodate a really new or unilual scientific ' 

idea, . . : (6)assess the inpact and make recormendat ions relative to' 
• peer review and provisions of the Freedom" of Information Act (FOIA), - 
' the Flederal Advisory Cormittee Act (FACA), and the Privacy Act (PA)." 
(Attachment 1 - Charge to' th^ Study Teaiji.) ... 

•.Bie Study Team was instructed to exanrine the 'pos it ive features *of the peer, 
review system as well as to assess critically any-def icijencies that might be 
. suggested: Overall, m examining the positive features, the Study Team feels 
that {Jeer review exercises the single most powerful influence on the continued 
high quality of the Nation's biomedical research effort and encomp^ses several 
fund^tal characteristics. These include the Selection of participants who 

according-to their expertise and a system of checks and balances 
designed to. protect the process from considerations extraneous to the criterW 
ion of high scientific quality. • - 

Furthermore, the Study Team believes JJiat confidence in the system is justified 
and has- Solved primaKly from, the steadfast position maintained by NIH that- it 
^^'^ foremost,- the high scientif i^quality of the supported research, 

as defined by t^ peeTf-review' system, whioiKshould govern the content of the 
• national biomedical research effort. Allegiance to this principle by a 
. succession .of program managers at nA, over a period of many years, has 
^ resulted m a research program not seriously .challenged in regard to its 
funaamental, scientific quality. / ■ " 

Uve^tiady Team further believes that the NIH research programs constitute a 
^ critical national resource, and that the quality of this reseurce has been^ 
. successfully defined in the public interest by the unique administrative dlTice 
of the. peer review systenf: ^ it is confident that the sysfcem will continue to-- 
provide such service with Careful- managemfent and maintenance and, an unswerving 
ccmmitment to the applicat^igi of the principles of scientt^c excellence. 

■ ORGANIZATION OF IHe'^'IUD Y * - . " • ' ' 

Ihe Team. recognized, that,- although as a group consisting primarily of 'NIH- 
officials* it could bring a significant collective perception of th'e'peer ' 
review system as seen from the viewpoint of tho^.who^w the system, besis 

" • . • ■ '27 • '• ' ■ ■ 
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ahd who are al^b responsible for* its qperation; nevertheless, its appraisal of 
the peer review system would be strengthened by input from the external 
oonAionity. Accordingly, the Study Team sought opinions about the peer review 
systan and suggestions l^r its j^rovejnent from the broadest possible range of 
bloroec^cal- researdiers and interested lay individuals other than scientists. 

To acccnplish the tasks necessary to fulfill its mission, the Study Team 
was organ iafed into a series of subconinitt^s. Eadi subcommittee was chaired 
by a fiienter of the parent Study Team, but the monbership generally 
included both Study Team members and other NIH staff , as well. Participation 
of the 'latter served to expand the r^resentation provided adcJed 
Expertise. Prfcress §uid con?)leted subcommittee r^rts were reviewed and ^ * 
modified as a result of discussion by the pareni Study Tecwi as a whole 
in order to assume concurrence and acceptamre-by all. 

Certain specific issues were assigned to single Study Team meinbers or 
small wwk groups in Order to develop position papers? These served 
as 3 basis for discussion and fdr incorporation into the .body of the 
report and as recommendatiorfe' of the whole Study Team. These discussions 
generally resulted in one or more revisicms prior to concurrence and/or , 
acceptance of the document. » 

A brief statement regarding each of the subcommittees follows; A more 
detailed description of the subcommittees' activities and. considerations 
will emerge from the body of the report. 

o Subcdnmittee on Studies of Peer Review Systems 

The Subcommittee examined' previous studies, of the NIB Peer Review- 
^ % Systep and also reviewed briefly peer* review systems in use k\ other 

federal agencies and other countrj.es. ' . - 

o Hearings Subcommittee - . " 

ihe Subcommittee prepared fpr and planned th?^ public hearings in 
different parts of the country to obtain the percept ions arid/or 
suggestions of the .external conmunity in regard to NIH peer review. 
^^ThQ Subcommittee also assessed, analyzed, and evaluated the information^ 
^derived from the hearings. 
* . . . ' . 

^,0 Survey Siabcommittee 

. The' Subcommittee developed a survey questionnaire which was used to 
. obtair\ opinions regarding the NIH peer review system from members of, 
all current Initial Review Groups (IRGs) and Advisory Councils 
attending the November 1975-February 1976 meetings, and subsequently 
assessed, analyzed, and eval'uatAJ the findings. 

" , , - *■ 

<5' Subcomnittee on Appeal^ ' - ^ ^ 

The Subcommittee evaluated tj^e desirability of establishing a drants . 
p^ review appeals mechanism, and, having determined the need fM|fc - 
' sdoi a' system, proceed to develop appeals procedures. 



o , Subconmitteft to Review Legal As^pects;^ Peer Review, ^ 

^ Hie Subcx3mfnittee examined the Ife^al cohsiderations related to peer 
review, such *is Freedom of Information legislation, the Privacy Act, 
ccHifidentiality, conflict of interest, chartering of .committees, ad 
hoc reviewers, and selection off committee members, as well as the~ 
impact of these factors on the peer review systero, . 

o Subcommittee on Business Managanent Practices «. . " ^ 

Ttie Subccninittee,,stadied the impact of business managanent practices 
on peer revie^yrantj review procedures related thereto. It also reviewed 
and made ceconinendations concerning the establishment of policies • 
^ regarding the role 'of a business pcfeultant in peer review.- 

Additional topics considered by the Study Team and dealt with in depth in 
♦.the body .of the report or its appendices were the following: 

1. Current procedures for selection of Initial Review Group, Special 
Initial Review Group, and Advisory Council members and modifications - 
of these procedures. ] ■ \ • 

2. itie effect of review^ workload, on quality of peer review, and suggestions 
to lessen the workload. . 

3. Summary, statements and priority scores. 

4. Pranature disclosure of the outcome of peer review. 

5. Ilie role of peer review ^in support of innovative research, 

6. The role of peer review in NIH decisibn-mdlcing, 

7. Th^^^^MratiOT of peer 'review from program responsibilities.' 

8. Tfie need tpr iajproved cortnunication between NIH and thfe external 
•community as well as within internal NIH components. 

9. An a nalysis and evaluation of * letter responses fr6m the external 
tewnunity conineriting on aspects of the NIH peer review system. 

IJie Study Team also- prepared a ccsflpr'ehensive desfcription of t;he NIH grant 
plication peetr- review pr6<^ss. (Appendix C). 
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If ATiowAL twrnrvm or MzArm 

TO s B/I/D Directors oate: April 28, 1975 

ECEA Members , ^ . 

00 Staff 

• • • I* 
noM : Acting &1rect>r, NIH ^ . 



nnuECT: Establishment of NIK Grants Peer Review Study Team 

> • * . . 

During the spring of 1974, the Executive eonmittee for Extr^tmural 
°. Affairs sponsored and participated ifi a semina«L.finj?eer review. The ; 
IMjrpose of this seminar was to identify and evaltjate the objectb^s 
of the peer review concept, document strengths* a*nd weaknesses of thg NIH 
peer review system, and discuss possible modifications or alternatives'. 
This effort derived fromHlhe dlial 'perception that NIH Staff, who know 
the system best, should be in a unique position for Its sppralsal ^nd that 
the* system should indeed be subjected to periodic In^pth evaluation to * 
. " assure its continued quality and effectiveness. * f ^ 

In addition to reyiev/ing previous studies of peer review and the 
resuUlTtg suggestions and related criticisms, the ECEA Seminar format 
. provided for a current appraisal of the Extramural Research* and Training 
Grant Review System and a re-examination of basic concepts and p^rocedures. 
This included a consideration of prepared papers followcTd by panel 
discussions and the ratification* of a set of recommendations "which were 
part of a final report of the^seminar submitted to ro? in December,- 1974. 

' ,^ ^ ' N * • ' 

Following up a major reconmendatloh in the final report from the 
Planning Committee f6r the ECEA. Seminar on peer review, I am establishing 
an NIH study team to extend the efforts of ECEA and conduct a detailed 
and cc»nprehensive study^of the'NlH peer review system. This study, in 
broad outline, should focus on the philo^pphy and procedures of peer 
review,' Its applicability to the NIH awjirdfi^g instruments and programs, 
/ . the attributes^and problems of .alterna):iv^, and the role and character 
df peer review in the decision-making process at NIH* In ad(^t1on to 
' this major" charge, I am asking that t\\e study team: . * ' ' 

* • Review previous afid^ on-going studies an^^^^ 
- analyses of the NIH and other systems of peer 
review, evaluate rscorrimendations and, in tne * ^ 

course of the v;ork of the stuay team. Initiate 
; additional analyses or activities as necessary 

for accomplishments of team objectives. 

• ^rtifculate^ in detail the philosophy, objectives, 
.procedures, and accompli shmerlts of peer review. 
From such descriptiocs/ the study^ team should • 
* ' ^ r prepare or have prepared one or a series of paoers 

for dlssemin/tion to the public and the scientific 
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' coonunlty.as a means of achieving a general uric^r- 
ttandlng and appreciation of a system wh'ich has 
evolved to Its :present 'state over a period covering 
!«' quarter of- a century.?^. ' '-^ , ' 

• Examine In critical detail the entflf process of peer 
review apd make, where necessary, rec^mnendatlons for . 
■odiflcatlons or change, addition to other items 
which' the study team may'wish to consider, specific 
attention should be given to (1) selection procedures 
ind criteria for advisory committee members, (2) the 
nee<|-and feasl-billty for an NIH polfcy defining the 
tdmlhlstratlve and scientific-relationships of the' 
Staffs responsible, for the technical merit review . 
of grant applltcatlons conducted In awarding Institutes 
and Divisions ^r^d those responsible for scientific 
prognun maiiag^mfen.t, (3)^ the need, advantages, and . ' 
disadvantages for a technical merlt.revlev/ "appeal", 
nechanism for appllcarrts and an assessment of the 
process currently In use, (4) the role and character 
of peer review in the decision-making process ati NIH, 

' and ^5) the .capability of th<peer review system to 
' acconinodate a really neW'or unusual scientific idea. 



Evaluate tlTe need for enlarging or alterl-ng the type 
of NIH staff training and communications to Imprave 
and maintain coordination and interaction between 
separate NIH peer review activities.. 



Assess the Impact on and make recommendations relatlvi 
to peer review and provisions of th^ Freedom of. 
Information Act, the Federal Advisory Committee Act, 

,and the' Privacy Act. . 

i'l . • _ _ ♦ \ , 

To acafcnpllsh the above -objectives and to consider other questions and 
recow^ndatlons concerning the peer review system' for grants, the following 
stikly team is appointed by this memorandum: 

Or .V Ruth Kirschstein Dii^ctor, NIGMS, Chalriwrson ^ 

Or. Robert Akers Pbllcy ar^ Procedures Officer, OERT, OD ' 

Or. Geor^ Brooks', Assoc. Director, E"Xtramural Program Activities 

Hr.yCarl FrettS ' Director, Division of Contracts and Grants, OA, OD 

Or. Norman D. Gary Executive Secretary," SSS, ORG 

Or. Willlatn Goldwater Assistant to t^e Associate 'Director, OCR, OD 

Or^ PhUlip Goi-den Acting Clinical Director; CI, NIAMDD 

Or. Jerdme Green Director for Division of Extramural Affairs, NHLI 

Or. Ann A, Kaufman Research Grants Officer, OERT, OD 

Dr. Vllltam Raub . Assoc. Director, Extramural & Collaborative Pregram, NEI 
Bir. S. Stephen Schlafffno Associate Director for Scientific Review, DRG 
Or. Katiherlne S. WiUon Executive Secretary Genetics SS, ORG' 



In pursuing its-work, the study team is requested to make full use 
of all necessary resources 'both within and external to the.NIH. Within 
this framework, 1t is expecte'h that appropriate consultation will be 
nade wijth members of the scientific and grantee conmunity and current 
and foraer members of NJH Advisory Groups in order to examine 
perspectives, criticisms^ and suggestions for, alternatives or improve- 
Mnts to the peer review system. The office of the Associate Director 
for Extramural Research artd Traini7ig*will make available to the study - " 
team the full report of the ECEA Seminar. on peer, review and other such 
documents. anB previous studies'that will be useful in its deliberations. 

asking that the study team report- interim activities to me through 
the Associate Director for Extramural Research and Training and that a 
final report with recommendations be sut^itted no later than May If 1976: 

I would greatly appreciate^all cooperation and assistance tha't you and 
your Staff can render the study team. In addition, if there are comments ; 
or suggestions which yftu or your staff feel may be useful to the study 
teaoi, please cogmunicate them directly to Dr-. Ruth Kirschstejh. 




Ronald W.' Lamont-Havers, M.D. 
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. STODIES OP PEER REVIEW SYSTEMS 




Apparently, peer review procedures for health research supported by the 
^e(3eral Qoverinment were initiated in 1902^ At that time the Fifty-seventh . 
congress est^fSTTshed a Scientific Advisory Board of non-<k>vernroental scien- 
tists to assist the Surgeon General in the administration of the Hygienic 
n^tqt^x>ry. Iri later years (1930) the Laboratory w^s renamed the "Nationatl 
institute of Health" and the Advisory Committee was reconstituted and | 
^renamed the "N^icwial Advisory Health Council." ^ - ' 

When the National Cancer /institute- was established in 1937, the Cancer Act 
also provided the legal ba$ik for the Naticmal Advisory Cancer Council. 
Ttxe Council wc© to play a key\pie in reconmending'tfie award of grants. 
These author it ieis and procedure^were extended to grants and fellowships 
in all he^th research areas in 1944, v*ien the Public Health Service Act 
was passed. 

In 1946 the Director of the NIH recognized the need to establish a more 
formal advisory mechanism for the assessment of technical and scientific 
merit; the Division of -Research Grants (DPG) was established and study 
sections or initial review groups came into existence irt 1947. Thus, at 
that early date, the basis for the current NIH dual review* system was 
established. r .7 

Over the years, since the Very inception of the dual peer review system, 
there have been nwltiple studies of the NIH advisory structure and the 
_ review and approval system. As NIH review procedures have matured and 
developed, andt as the extraimaral programs have expanded, these studies^ 
an(^ surveys have become^nore frequent and extensive. Sorae were instituted 
by the Executive Office of the President, some by Congressional mandate, 
and some by the Departmoit of Health, EkJucation, and Welfare (HffiW) or 
the staff of ^ NIH. ^ The members of the Grants Peer Review Study Team (GPI«T) 
have cataloged and reviewed these previcHis studies. Some focused in 
particular on NIH while others were of a more general diaracter deaHng 
with^ overall F^ederal policies^ Those studies th« seeirpartici^arly 
relevant to the current undertaking are briefly reviewed here. 

In 1947, the same year that the MC wag established, the President estab- 
lished a scientific research board to study and report on Science and 
Public Policy. Volume's of its teport is entitled "Ihe Nat i<xi^s Medical 
Research. "(1) Ef^en at thxs"^^afiy stage, the Board noted that the use of 
advice from outsi^de experts on research proposals presented a difficult 
dilenina; While the selection of ^xper£s from the leadihg hospitals, medical 
schools, and research' institutes insured the best possible advice,' on the 
other hand,, as the Board noted, the majority of. proposals to be evaluated 
were submitted' by staff members or graduates of these same institutions. 
It' was concluded that the advisory groips were providing splendid service 
to the Government axjd devoting conscientious thought and considerable 
time to the review and evaluation of the proposed projects^vrfiile resisting 
.any tendency to create research monqpolies. The Federal Government wa6 \ 
advised to continue its sound policy of utilizing outsibe scientific advisers. 



In the inid-1950s, the Secretary of DHEW coninissioned a report on "Ihe 
Advancemait of Medical Research and EkSucation Through the DHEW,** niis 
report, cdnmonly known as tH^ "Baynes-Jones Report," was. issued in 1958. (2) 
The. consultants reconmended the expansion of medical research and educaticm, 
andtprojected that total naticml medical researdi expenditures should 
increase frpm the 1957 level of $330 million to approximately $1 billion 
per year by 1970.* The report^ briefly described the Nig peer review system 
mS noted that the advisory structure and procedures were designed for a 
irelativel^ Ismail grants program diaracterized by grants of modest size.' 
The report predicted that the NIH extramural program of research support ' 
would evolve towafc^ larger grants for broadly -defined purposes enponpassing 
many disciplines arnj would require changes in the traditicml study secticxi 
system. 

In 1960, the Senate Ccnnjittee on Apprc^riations requested advice'^from a 
cqmnittee of consultan£s on Federal^ support of medical research. There 
is no, indication (in this so-called "Bo Jones Report") that the appointed 
coirroittee made an ind^Jth study of review procedures. However, in deter- 
^mining that funds apprc^riated by the Congress for the support of researdi 
on major disease problems had been escpended by Federal agencies with 
remarkable efficiency, the Ccirmittee report iopluded the following . 
statem^t: ' ' o , ' 

"Ihe NaticHial Instit$^s qj^ Health, ^ich CCHigre^ has made 
its principal instrument for the disfcwrsement of funds for 
medical research, has developed an extremely successful system 
for teview and appiroval of applications for research gi^ants, 
training grants, and fello>*ship^ Ihis plan, relying on close 
supervision of outstanding sciertists, has assured, ocmts is ten tly 
high standards for the research supported, gained the confidence 
of the scientific coinmunity, and jnaintained the-±raditicMial 
freedom of both institu^fions and investigators." (3) " ' 

Many similar statements were made in the November ld60 Report: of the 
President's Science Advisory Comttittee (BSA&). "In its, support of t>asic 
research, the government has usually relied on th^ advisory judgment of 
respected scientists, and in the main" this advice has ensured that: in 
those areas of research in v^ich Federal support has been available, out- 
standingly^ have been able to attract substantial support, In this 
i/espect, the^ project method of research support has real values lAich 
should not be forgotten...." (4) ^ ^ ' 

^e Coimittee on Science and* Public Policy of the Naticml Acadeny.of 
Sciences (under the Chairmanship ^of Dr. George B. Kistiakowsky) issued a 
report ir*i964 concerning "Federal Support of Baaic R&ieardi in Inst:i- 
tutidfis oil^Higher Learning." (5) This^eport gives a history of the "origin 
and the developm^t of the Federal scientific estiablishmait and (jiscusses , 
the use of grants and contracts^ and peer review procedures, in a variety 
of Federal agencies. The studyiiad its origin in ^ resolution passed by 
the American Society of Biological Chemists which requested an examination 
of Federal programs in support of fj^ndamental research. Ihe Committee 
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expressed 8i()port foe the panel system of peer review. However, disadvan* . 
tages of* the panel system af peer xeviev^ were mentioned, including the 
. %iorkload accepted by scientists who partici^ted, the increased time to 
^cxnpl^te the review of each proposal, and the potential difficulties v^en 
service on peer review groups is concentrated among too few individuals 
or institutions.* Ihe Cooinittee also noted potential problems with* inter- 
disciplinary researdi, namely, that panels jmay not alwa^ reflect the - 
'current frontiers of research, (x that the^'proposals may be s6nt to the 
wrong panels. Recommendations .for th^ inprovem^t df panel peer review 
were provided: (1)^ the use of more advisors^and more frequent rotation; 
(2) the use of youog^r scientists ^ panelf; (3) the inclusion of perscMis 
pf breadth as well as specialization; the addition of members from allied f 
fields; (4)' tjhe serious <xxisideration of the advice of panels in order to 
maintain the conviction among advisors that their services are important; 
(5) the, fact that overall budget review is appropriate to peer review; 
but detailed decisions on fiscal matters are rwt; (6) continuation of 
support of /basic ^ research by several agencies so that scientists will have 
access tb m6re than one peer <eview process. j 

In that ame year , 1964 , the Select Committee Government Research of . ' 
the Hou^ of Representatives < the Elliott Committee) issued its.rqport on 
natitiiyD goals and policies. Ihis Ccmnittee was established to provide 
the Bojser'of R^resentatives with an overall review pf Federal research 
and development progums. Althou^ none of the t^ studies reported dealt 
primarily with NIH or with peer review, » the Committee examin^ various 
methods used by Federal agencies to review applications for si^^port. ihe 
Connittee concluded that fellow scimtists must rate both the conpetency 
of those presenting prcposals and the ripeness of the f ilsld but thit the 
evaluation cri€fe«4^ used must includ^ value, economic, and pdlitical judg- 
ments, as well as Btoader judgments concerning scientific and tedinical 
matters. The Ccmnittee noted that the panel systems utilized by various 
ag^Kies provided a training ground for the developitent of a pool of 
advisors required at higher policy levels. (6) 

Perhaps the largest^ most extensive ^ and^most prestigious study of NIH was 
the one commissioned by President Kennedy and chaired by Dr. Dean Wooldridge. 
Ihe Hooldridge Contoittee, supported by twelve working pa^lsr submitted its 
report on "Biomedical Sciences a^d Its Administration^ in 1965. (7) Detailed 
^descriptions and analyses of the NIH peer review procedures are contained 
in yarious'^sections of that report, but particularly in the report of the 
Pariel qp Review Procedures. Ihe first and most importiirit general conciusi<Mi 
of the study was that the activities of NIH wece-essentially sound^ and that 
funds were being spent wisely and well in the public interest. 

"The opinion of thte (Wooldridge) Committee, based on the 
, exte^ive investigatiixis of its consultants, is that the 
large majority of the intramural and extramural research 
supported by NIH is of high-quality. We strcxigly ^sprove 
the peer eveduation method of selecting recipients of 
extramural grants." 



' 5TSie Study Section Procedure utilizing scientific peer judgmeits 
^ is the best available' method for airarding research grants." (7) 

It is believed that^e Wooldridge Coninittee conducted the most caipre- 
hensive and thorough review ever undertaken of a Federal science agaicy. 
Addressing itself specifically to the extramural programs, the Ccranittee 
felt that despite increased loads on study i^lctions, peer judcpnents should, 
be preserved at all costs. Measures should be ei^ccwraged which would 
Jessen the burden on* study sections without usurt)ing their function of 
scientific review. The Cormittee felt that extramural investigators would 
rtJ^more contact with 'and advice from scientists on study sect i<»is and 
would welcome more site visits, suggestions from study sect ions *about ^ 
improving research^plans, and more explanations of why^roposals were not 
approved. v 

In 196^f no fewer than three important report^ were published.. Ihe Westrate 
Report, connissicM^ by the Bureau of the Budget, dgalt w^ith the management 
0f Federally funded research programs in several areas, including biomedical 
sciences. Information was ctotained by interview and questionnaire roethoSs 
,that involved Federal officials, university staff members, and officials 
at private foundations. Hie. section. of the report most relevant to review 
procedures is entitled "Improving the|puallty of Federal Researdi Adminis- , 
tratiOn" and portiOTS of this deal specif ically with HIH methods, viz., 
"thfe most elaborate 4: ev lew procedures of ar^ ag^cy." (8) 



The report recomends that: 

"/^rc^riate action should be taken to remove the statutory 
provision which prohibits the Surgeon General from authorizing • 
approval of an NIH grant unless the grant has been expressly 
approved by the Advisory Council of the pertinent Institute. 

"With respect to prqpbsal review procedures, the agencies should: 

a. continually evaluate the adequacy Of the advice they obtain 
from proposal reviewers and the procedures they 6nploy; 

. . rh. j^ovide sufficient administrative assistance arid^ numbers of 

advisors to reduce the reviewers' vjorkloads and. insure that , 
* , their time can be most profitably utilized; 

c. because of the advant^ages inherent in p^mels, use panels for ^ 
Scientific review to*the maximum extent feasible; and 

^ d. when panels are used, rotate the membership and give ccxiside- 
^tion to geggraj^iic and university distribution in selecting 
appointees. 




^ "Xn ack3ition their present procedures for dealing with possibilities 
of conflict of interest, 'ag«lcies should: 

a. stress the advisory na^re of panel deliberaticxis, and instruct 
the'Goverriment manber present at such panel sessions to deal 
with potential ccmflicts v*iidi may arise. 

b, bar a consultant or panel member from participating in general 
^ discussion^ about proposals sulGoiitted from any source in his 

own university, although he should be permitted to answer 
spsQific questions of fact regarding such proposals. 
• 

"As part of their review procedures, all agencies ^ould request 
panel or consultant opinions regarding the general level of funding, 
major equipment requested, and fraction of investigator effort 
estimated for the project when such opin^ore afp^ar to be relevant * 
- .to deciding the merits of the proposal. ^ i 

4 "Agencies should rou1?&nely distribute an aw^rd list to their prc^xssal 
referees." (8) - * 

The report of the Secretary's Advisory Conmittee on the Management of NIH 
Research Contracts and Grants (the Ruina Commi^ttee) was also^^^lished in 
March 1966.(9) Once again, the review procedures utilized in OTfr^«s^eunaral 
programs wer^ examined and endorsed. 'Hie report of *the Ruina Coram it^^""^"^-^ 
'Stated that:* 

"With the enormous expansion of the grant activity during the 
past decade,, the Advisory, Councils have^ found it necessary, as 
a pVfictical matter, to rely almost entirely upon Study Sections, 
panels of distinguished scientists organized on the basis of 
scientific disciplines iftedical specialties, to evaluate the 
"scientific merits of graJit applications. So large a volume of 
*eqpplications is now processed" that Advisory Councils are almost 
entirely, dependent upon summaries and "priority ratings" forwarded 
by the study Sections. ^ . • ^ 

"Ihis inevitable dependence on 'peer grotqps' for preliminary ^ 
screening and ordering of grant applications has not resulted ' 
in a mere 'rubber stamp' function for a Council. Rather, policy 
deliberations undertaken by a Council are often based upon 
questions cc»icerning individual grauits whidi are singled out by « 
notations of the Study Sections, or by NTH staff or members t>f 
the Council at the time of meeting. Eadi Advisory Council 
member receives summaries of all of the appllcatlcMis for grants, I 
and thus has a convenient reference file of current information 
to assist him in keeping up with the broad trends in research 
pertinent to his Institute's 'field of interest. ^ 
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*The effectiveness of these project grants in support of biomedical ^ 
research is widely recognized, as is exemplified by the following ^ 
CGninent of one of the panels of the Wdoldridgaf^Corailttee.' * ^ • ' 

* "Ihe procedures followed in reviewing v*iat are knom as traditional 
researdi grants were institutjBd first \an3 constitute the most 
thoroughly established, the most rigorously follcwedy the::easi^st 
to understaild, pr<*ably the most widely admired of ttte activities 
included In the extramural programs of the National Institutegjof 
Health. *r (8) , < 

Later, in 1966", a -special Congressonal subconmittee chaired by Cciigress^tnan 
Paul G. Itegers investigated DHE»^. (KT) With rfespect to the functioft and . * 
adequacy. of the Advisory Courtcils, the Subcommittee endocsed the - ' 7 
criti^e of the I^uina Corani^ttee. The Subcommittee noted that the NIH^ • 
system of review and approv»#Bs ccxisidered desirable by the scientific . 
community primarily beclause Sr6posals«wei;e independently judged by peer^ 
and not by Government officials, ^e JIIH grant review system thus sepa- . 
rated the asseiferoent of scientific merit from the staff respoRsibility 
for program administration. Jn the opinion of the Suljcomnitteer one of 
the weaknesses of the NIH sysben resulted from the fact that it w?is devised 
■ for a nwch smaller volume of grant applicatiafe; it was noted that it was 
virtually impossibly for review groups, including the Councils, ,to conducjt . 
cortpletely satisfactory review in the limited time availerf^le. _ 

Among other things the/Rogers Subcommittee stated that: , 

The increase/in the^number of study sections had not kept pace 
with^ t;he' inwease in the mimber of applications; the plosion/ * 
of scientific knowledge as a result of increased eraphasis 
placed on research; the unrealistic assunpt ion, that the 
confcined knowledge gf the members of study sections and councils 
will in each case include all thfe latest informatioh aval,lahie^«/ 
in a particular Specialty; consequently in view of the latter, - 7^ 
recormendations^r approval or disapproval ^^speat gener^aiy * 
to be the judgment of -tsc^ or at best two peers. It waa 
recommended that consideration shjjttld be given to revising tiie' 
systOT for review and aqppfoval of^grant* applications: in order ; 
to better cope with the lar^e volume and high degree of ^ ^ 
ccnplexifcy of the plications. This Report also pointed^up; ^ 
the fact that, certain questions raised by the Wooldridge • 
Committee stii\ remained unainswered; .namely, does the s€udy ' ; 
sectic^ discussion modify a priority rating based only on 
• reading the application pri<^ to' discussion; v*iat detecminea 
^ which applications are site visited;^oes the site visit . 
medify an original pri9rity rating— M ^ic4i directionV • 
how ia the quality of study section decision Evaluated? (lOJ ^> . 

In 1967, the American Medical' Association's Commission^of Research delivered 
its report. (11) Many of the reconnendations of this Cqninissioa were aiJdrMsed 
roecifically to the American Medical Association. The Cotroission did ' 
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conclude vthat thi study section mechanism of NIH had been successful in 
idenl^ying aaxl, supporting high quality research. The Cormission reconnieinded 
ti)teit pntership on study sectixms- should be rotated to prevent the develop- . 
/ aent of * an inbred, elite, decision-making establishment. The report also^ 
. dj^Sw^t^ that the primary consideratiOT in selecting monbers of Advisofy 
Councils should be profess icjjal attainment and. the ability ^tp make unpre- 
jjudided, statesmanlike judgmSlts. The Cfinmiss'ibn also recomfioided the 
blii^iment of an advisory .gro^p for tSie Director, NIH. 




^Sie Pamtain Comit'tefe Reporl^in October 1967 (12) ,examin^ "The AdnfiniS- 
tratiwi of Research GfSmts in the Public Health Servic^" The report ^ 
focused iqpon the principal, researdi arm of the KB, the NIH. (It should 
be recall^ tiiat^the Coranittee had issued previous reports on this subject 
in 1961 ai^»1962.) The Cormittee was cc»icerned aboqt the -g^. in NIH 



resfeardi siflpport betweeij|^|tt "rich" aiW "poor" sdiools. One reason given 

that a relatively 1 flHWcJ * nimtoer of institutions furnished the bulk 
the PIB c<»isultants and it was felt likely that these advisors reacted ' 
more favorably to the institutions and scientists that they knew best v^^ien 
evaluating project applications. Moreover, study section and council 
menbers were in a unique'^position to learn of research 'opportuniti^^^^^f 
their ^ia.Jds aira to share with their colleague? ah ihtimite knowledge of 
how the ga;5»t., system c^rated. A ^ond 'concern was the tendency, in PHS,^ 
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to appoint a anall group of indiviOTrals to multiple terms gn councils and 
coninittees and the appearance of favctitisra v^en thes^ same individuals 
received substantial NIH grants. The;Comraititee reconinended that each 
oounbil appointrooit J3e limited to one four-year t^rm with members being 
^ ineligible for reappointment for a period of fopr ye^s. It also jFeccm- 
mended that cohsideratiorf be given, in the Selection of committee raemberd,' 
*to balanced representation of geogrc^Jhic^ regions 'and educational institutions 
and, further,/ that to the extent possible ^ consultants shpuld be drawn from 
among qualif ied sc lent i^tsr who are not themselves cecipient^df PHS grants. 

In February 1966, the DHEW issi^ed a response to the Fountain Cormittee 
.Report. (13) There arey in particular,' two jsect^l^lfts of this 98-page report ^ 
that are especially relevant to the interests of the current NIH iSrants Peer 
Review Study Team^ The first discussed the review system'at NIH and ItS/ , 
roie in mainftining the high .quality of r&earch that was belr|L supported. 
In the section orgjedvisory committees, recommendation^ were^*afc in jegard 
to limiting tiie p«ioA of appointmoit of NIH advisors, geographical - ^ 
distribution, and. to%te development of standards to insure that ^j^eritists ' 
of hi^ quali^{^re '^pjwihted to advisory groups. , - 

In*197(l, *he Council of Acacjemic Societies of the Association of. American 
Itedic^g. Colleges (AAmC) undertook to fw^lat^:^ A Policy for. biomedical. , 
Research. "(14)/. Ihe Cdnmittee focuspd particularly on issues relating to the 
d^eloproent of bionediccQ researdi raar?)ower. 'fte repc^€ supported 
cent ii^ed entasis on th^ individual project grant aw?irded through peer v 
re^i^tew as the primary instrument for the support. of biomedical research, ^ 
alphg with an expandetJ system of program project support addressed to ^ * 
E^c|^ems of special relevance., "Hie report included Strong eridorsemait 
of peer review but * stated th^t r^eview mechanisms shou[Jd be streamlined, 
Ihere were, however, no detailed diseussions*br sproif ic recommendations. ^ 



ERJC: 



dh i ti'ig the past several years^ there have been increasing inquiries into ^ 
the.NIH peer review system. Sane of these have followed upon the piassage 
of the Freedom of Information Act (P.L. 89-487) and the Federal Advisory 
Comnittee Act: (P.L. 92-463), an^ have stimulated a general review of 
adv.isbry group, act ly it ies and , functions within the Federal Goverranoit. 
Within DHEW^ additional controls and ^requirements have now been estab- 
lished with respect to the public advisory groups and these apply to the 
;Nip public advisory groups,. Thus, pjpcedures for the technical merit 
review of scientific research proposes have been modified. 

Among 'the studies and surveys that have fed to such modifications, one . 
must include that of the "Coqper Cortinittee.''{15) Ihe report of this NIH 
Progr'arti Medianisms Committee was issued in February 1973 to the Director, 
NIH. In this report, ^the Committee included' seven recommendations bearing 
upon th^ grant and contract programs. Especially noteworthy is the 
reccnmflndation **that NIH establish and refine the use of uniform policies 
and standard procedures by \i^ibh its ccnponents define, develop, and 
implefnent programs, and initiate, review, select, and manage projects 
funded by contraclft, grants, and oth^r awards. "(15) 



In April 1973, NIH an^ the Health Serv^ices and Mental Health- Administrtrtiion 
(HSfflA) received an issue papfer pr^red by the Office of MaSiagero^t and 
Budget entitled **1he NIH/NIHH Peer Review 9ystem.*'Jj^^ Hie pager contained a 
^^perally^ accurate,* brie^ description of !^IH/KlMH«^* review, including its 
statutory bases; In .-its identification of problems^th the peer review system 
and its formulation of alteri^^iveS^ for inprovjj^g thPsystem, however, the 
paper displayed several serious misunderstandings and biases. * Nonetheless, 
"the repor^^ and the commentary prepared by; the Office of the^ Director, NIH, . ^ - 
are of considerable interest. , Both documents were considered and difscussed - ^ 
in the July 1975*special oversight hearings of Naticxial Science Founclatlon 
(NSF) peer review (House Subcopnittee on Science, Research, and Technology). ^ 

Uie requirement , in. the National Cancef. Act /ynendments of 1974 (17) for scientific 
peer review of applications for bioroedii:al ^nd bdi'avi^al researc^apparently ^ 
rescltted from the hear ingsi before the Sybcommittefe, particularly ^ r * ^ 
. prepared statonoit from the AAMC. In urging peer review foq contracts, ^ 
the AA»^ stated that (1) the use of nonfederal scientists to xeyiew gfant 
. applications for bi<?m^ical reseafch projects to be funded, ^^^ihe Federal ^ * 

Government' h^s assured a broad, nonfederal voice in the forroatibn and ^ , 
. inplejnentaticm of national policy; (2) the peer '^review systCTi v^ich has 
^beeh developed and utilised for NIH grants has assured a rigbrous asse$s-^ , 
•meit of ^ the scientific merits of-research projects for vrfiich ,NIH grant ^ 
^.^Ijupport is being sought; (3). this process ^of scientific cppr a isad carried^ 
fC&t by disinterested aihd expert: scientists has resulted in the use 6f ' 
public -funds 'OTly for the support of bicroec^ical research which has ^net • ^ 

' ;)ftl|(^lghest standaj^/Of es^ellence;^ and (4). th^tjprocess of peer review has 
beeo' recognized a*wi effective mechanism in'assuring maxijnun^cientific * 
returns for public investments in researdi grants. ^ ^ 

. In one of the few studiea dealing with the effectiveness of decisibn- . • 
. ^ making in providing research support, ^Grace Catxer o^ the Raaid CocporatiOT^ 
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de\^lop6d an original and rather ii^enious, statistical method to examine 
NIH managesierit and peer review dedKons. Measures of research output 
and quality were developed and these were then, used to explore trends in 
ttai^ quality of grants-supported reseeurch and tJ^^^P^rcTtion of the NIH 
peer'revieit syst^." This 1975 tqportfV entitled^ ''Peer Reviewr Citations, 
and BionediW Researdi Policy: NIH Grants to Nodical Sdiool Faculty, ''(iSJ 
is of consic^rable interest and value,. , ^ ^ 

It is notewcxrthy that the NfH, in Mardi 1975^ submitted a report (19) ±o the 

'then existent President's Biomedical Research Panel concerninq the s^tus 
of inplBnentatioh of the reccnin^atifms of the Wdoldridge Conmittee. The 
reader should recall ,that, in Febr^uary 1965, the HSdldridge Committee (7), 
establi^ed by the Presid&nt, made eighteen reccranendations. Eleven of 

•these were acceptecl in v*iole or in part and iitpleroentation was begun; 

i^even reccnqiendations were not acc^ted. imyn^ the recomnendations that 
related directly to^peer review, there was a statement "that Advisory 

^Council mesDbers owe' their appdintnfent to no hi^her-JSovernment level than 
that of the Director, Nm." The NIH noted that thiS' recorotitendation had' 
not yet been iraplerooited. The Wdoldridge Conmittee Asp recomnended - 
"that the study section procedure utilizing peer judgments should be" 
preserved and str^thened and that the wQ^rkload on individual study 
sectiontfiembers should be lessened."(7) The NIH status. r^xDrt indicated 
that this hdf} been implemented initially but was affected by later changes. 
The workload of consultant reviewers has been seriously aggravated and the 
numbef^of wmmittees has been .sharply curtailed. It is noteworthy that 
the average workload 6f an NIH Study Section has Ancr^ased remarkably: 
from an averfge of 55 grant applications per meeting in 1^69 to an 
average of over 90 applications *per meeting in 1976. 
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r ■ OTOER REVIEW SYSTEMS IN t!hE FEDEIRAL GOVEBMMBn' • 

. ■ ' ^ ) 

Hhe NIH Grants J>eer Review Study Tteam also undertook an examinatiwi of the 
review procedures ,in several other Federal agencies; this was conducted in 
the fall of 1975. /Irrtwo agencies tiie review procedures appear to be 
e^)ecially relevant t;o the NIH. A btief suitmary of re\^iew procedures and 
policies in several other Federal agfencies is qiven below. 

1. THE NATIONftL SCIEMCE gOUHDATION (NS^ )^ ' ^ 

There are five separate directorate's at the NSP and there are, as a^ 
consequence, five separate review systems. Uiese are described, in the 
June 1975 NSF Staff Study on Peer RevieW and Proposal Evaluation. 
In general, NSF utilizes three distinct onions: panel reviews, mail - 
reviews, and :staff reviews. On the average, for all of NSF, d proposal 
ife reviewed by 6.5 reviewers'. Ihere is no single list of NSF reviewers 
, but lists of panel roenibers are available. Forty-four percent of all 
oonpeting propqsals receive an ad hoc review only. Twenty-ei^t percentr 
of all proposals, are reviewed by a panel ionly. Some 28 percent of 
proposals receive both panel and ad tKx: review. Some proposals submitted 
to NSF are reviewed solely by stafF.~^ese, however, are a distinct 
^ minority and represent sucii activities as cohferences and workshops. . 

IXaring 1974, the NSF received approximately 120 ,000 reviews. (In 
^cxmtrast, NIH reviews about'12^000 applic&'t^ions. ) . * ' \ . 

In each of tJieNSF directorgt?s, it is tfie' program officer v*io is 
responsible for formulat'ing a recatromdatibh^ tfie sfelecticm and the, 
funding decision are made at a higher edition.; The program officer 
makes his Teccnnvendation ta a Section hea(J^):>ufe often thiig is referred 

' on to a divisiori head, assistant direct^ &r even^to the Naticxial — 
Science Board. The Board reviews those g^roposai^ that are favorably ^ 
reconinended by a panel' and' that involve mc^ than half a million 
dollars per y4ar or a total cOBmitment- in excess of two million 
dollars. Ihe Board does not review proposals recommended jpfavorably. 

I * *^ ' -J ^ ' * » *■ * * ' ' 

Recently, the tJational Sciende^ Boafd^^passed a resolution indicating ' 

, that* full lists ofc. reviewers shouldHiDe published awl that verbatim 
copies ofi r^iews, withoilt t^e reviewers*, signatures, should be ffbt- 
warded to^the applicant invest igaioi^^; NSF is currently informing 
reviewers that their vefbaftxhi*^ oortments will be made available to 
applicants. ' ^ ' . .-^^ 

Panels rate proposals that tjiey have reconmended favorably in five 
caSfegories from eaicellentjto fioor and these roughly correspond to the 
NIH numerical rating systqp of 1 thr<5ugh 5.» Staff provides^eedback , 
and reports to the panel at subsequent meetings concerning actions ' 
that "have been taken on thei» panfl,rfcortin«da^ions. Although NSF 
- staff men*)er^ have not dope so irf the pa^t, they will now prepare 
suramary statem^ts hrd pr6vide resumes of the panel assessroaits 



and l:ecann»)dations so that this informatics may be'releasjBd upcm 
apE^ropriate reqpoat. A focmal procedure for e^^als is being develop- 
ed at NSP but it is not expectedfi that It Will be used extensively. 

Within the Biology Divisicm, some 4,000 pr<^)08als are r^eviewed each 
year} approximate!^ one-third are renewal applications. About. 
30 percaat of all conpeting pr(:^)osals are funtJed. Currently, some . ^ 
35 percait of the fundib available are used„to award coomitted, non- 
ooopeting continuation grants. Within NSF, it has been decided 
(administratively) t|iat non-conpeting cc»itinQations sl:K)uld not utilize 
more than^^ percent of the available funds; this guideline has been 
set in an atterapt to maintain flexibility -for the support of new * 
ventures. Ihe maximiDn duration of grant support in the past has been 
limited to two years but it has now been extended to five. Therefore, 
the average duration of NSF grants is gradually increasing. 

A visitlpg ccninittee evaluate each NSF progrann area and ""the performance 
of the program officer, and provides advice to the Division Director 
and to the Assistant Director of the Foundation. It is of some Interest 
to note that about 50 perceit of the program directors at NSF are there 
on a rotational system v^ile on leave from their parent instituticms. 
These people are selected by the Divisicm Director or the Assistant 
Director, NSF. The NSF staff believes that this system has worked^ 
well* Of the 24 scientists currently in the Biology Division; 12 
2u:e there on a rotational basis. * 

/ , ' ' ^ 

Within the Biology Division, the averageyrese^grch grant amounts to 
sane $34,000 in total direct costs. There are, however, approximately 
40 larger grants, somewhat similar to NIB program project grants, that 
range in size from $250,000-$500,000 per year; most are near th^ upper 
end of that range. Bie total expenditures for the Biology Division * 
in support of res^ch are_7& mill ion- dollars per year. 

The average panel used by the Biology Divi^icNi has eight members and 
there /ire 12 pemels. The panels know in advance the amount of funds 
available for eadi program area and this information is taken into ^ 
account *ih the j^opisal- review sessions. Proposals that are reviewed 
are either furfded or declined. Thus, there are no approved- prc^)osal& 
that xsmaio unfunde(3. This is a feasible and effective proc^ure. for 
the NSF because its e^ropriat ion. is obtained early in the fiscal year 
and it is therefore able to plan ^cpenditures in advance^ 

*The review panels melt three tim^a year, generally for two days. 
The average :mjBnber of proposals reviewed^ per meetijig is 100. Each^ 
proposal is, on the average, assigned ^to four reviewers. Site visi^ 
are conducted but not frequently. Titavel funds are soroev^t limited^ 
oftentimes a site^i^it will be performed by staff members only. 

Ccmsuljtants are paid a fee of $75 per day but do not receive firty 
remuneraticm for homework or mail r^iews. 
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panel *Hierai<&rs are appointed for a cwie-year period v^ich may be 
renewed two tiioes. Ihus,. most cc^ultants serve a three-year term. 
In the Biology Division, at the beginning of ea<± panel meeting a 
listing of the proposals and the ratings* given by each of the panel 
nenbers is distributed. Ihus, , with' this information available, the 
^ greater part of the panel .meeting focuses^ discrepancies in 
ratings. *The Program Director serves as chairman K>f the panel; ^ 

There are no formal deadlines fo^ the submission pf applications to 
NBP, In the Biology Division therfe is an average lapse of four months 
\rfrm the receipt' of a' proposal to^the time that a decision is made 
and funding is made available. . 

The* Division has a small conti^ctjp^C^fam^cmd issues "Requests for 
Proposals" in the usual mannerV Contract propo^s are reviewed by 
ad hoc coranittees. T 

There are four dianges that the ^EF staff would like to implement: 

(1) to increase the number of panels for initial review; ^ 

(2) to decrease the panel worklo^ to no more than 60 
proposals per meetingi 

(3) ^to perform more project site visits; and 
\ (4) to have more staff available. 

* * 

The NSF is planning to .conduct several studies of its review processes. 
For e;an|>le, the handling and fate pf 1,000 ^llcaticms will be 
studied. In addition, a survey (guesticxmaires and interviews) of 
several hundred applicants and revjjs^^rs will'^tee-conducted to determine 
their views of the NSF rev^ew-s^tetu^ 



THE VETHOttB MMISISTRftTIOM (VA) 

The VA system for peer review of medical researc* is essential ^ 
intra«Bural system. There are some 135 stations in the VA system where 
research is conducted. Ihus, this is basically a dec«fitrali2ed intra- 
mural medical research program. Seventy-five percent of the budget is 
devoted to that decentralized system whereas the remainder is caitraliz^ 
and involves cooperative clinical trials and career developrooit awards. 
Centralized p6er review is required if more than $25,000 per year is 
expended, in a research project. 

Initial review is conducted centrally by merit review boards and the 
secondary level of review is conducted by VA, administrators. The - 
roenbership and nature of the second level of review have been developed 
to reflect the intramural nature of the research effort: difficulties 
with staff and employees in field stations are avoided; orderly 
terrainaticm and phaseout support can be provided readily; .and morale' 
problems can be minimized or resolved^ - 

Last year a total of $87.7 million per year was spent for roedicai 
researcii and $68 million of that amount was for decentralized research 
studies. . ^ 

An attempt" is made to fund all approved proposals. to some exteint. 
Furthenaore, t6 preserve local allegiances and authority, funds are.^ 
forwarded to the hospital for the support of the investigator's 
research but these funds can be reallocated by the local research 
committee. ^ . -i 

The procedures for peer review have varied somev*iat with time. 
Initially, boards were established and Review wajf ccxriucted centrally. 
There was then a brief period \*en there was total decent ralizatitxi 
of the review and decision-making processes* For the past three amd * 
one-half years the current system has been in operatitxi. 

^Currently, there are 14 merit review boards and ea<*i has 7-^13 
members. The boards ^e disciplined-based and they meet twice a year 
^for one t6 two days. ; Members have three-year terms. Each board has 
,at least one VA member, but -overall, 25 perc^t of the total member- ^ 
^ship are VA enployees. Many of the individuals seizing on these 
merit review boards are, ooocurrently, NIH study section members. 
There are two cycles per year for review with December 1 and June 1 
deadlines. 
. t . 

The 14 merit review boards are serviced by six staff members: three 
executive secretaries and three assistants. It is apparent, therefore, 
that one of the executive secretaries and ope assistant serve five ^ 
merit review boards. ^ 




A propbi&al may be reviewed by niorfe than yone review board at the same 
tijfte. IV^ity-five to'SO proposals areyreviewedhat each meeting.- A 
numerical priority system is utilized with a score of 10 being the 
best and 50 the poorest. The average/score is 27? any score i^u^jerically. 
greater than 40 indicates that fundiiigi^is most uncertain. Priority 
scores are derived j:>y averaging indjrvidual scores voted by the members; 
the^ are not "normalized."^ 

Each propoeeU^ is reviewed :by af^ast three reviewers^ one of whom, 
must be a member of the board, 'mis, many outside pinions are 
obtained. Each ad hoc reviewer is contracted by VA staff before the ^ 
proposal is sent to him.. It is o^^terest to note that the evahiations 
.and opinicxis submitted by the reviewers are tran^tted verbdtinr (but 
iwithout the signature of the reviewer) to the.hokjpital that submitted 
the (HTOposal and thus to the app^icark institution and the investigator. 
Eac^ applicant receives at^ least three written evaluations prepared by 
individual reviewers along with the overall evaluation of the review^ 
board. ^ 

Approximately 35 pefceit of proposals are disapproved. Bie siminary 
statements of the merit review bpard are written by a board member. 
Site visits by the board are not conducted frequently but if they are ^ 
held they are generally small with a member of the VA staff of the ' 
cettral office always in attendance.. ^ ^ 

Recommendations made by the review boardmare siroUar to those made , 

by NIH Study Sections in terms of dollar anxxints, 4uration of support, etc. 

The review board meetings are closed althou^ announcem^ts are made 
in the Federal Register . - Any public oi 
are referred to tne local VA hospital. 



in the Federal Register . -Any public or sciaitific inquiries received ' 
3 tne local VA hospital. ' ^ 

Ihere is nb formal appeals mechanism. It muat^be recalled that this 
entire review system is concerned with uitramural research activitijes. 
Thus, a disappointed applicant must have the agre^i^t his local 
hospital research carmittee before he can submit a new ^r revised 
request to the cmtral office. The local research committee oftm 
has many academic scientists from the medical school/university. ' 
For that reason local review i^--a. significant phase of th6 ^activity. 

Rfivjfli^rs assisting by mail in this process are paid. $^ for each 
opinion. .JSenbers of the merit review boards receive one day^ 
ccmpeosation for their preparation time. 

It should be noted that the peer review system described above is 
not universal* throughout the VA. Cooperative clinical studies are 
assessed by an entirely separate procedure. They are reviewed and 

•^budgeted through a differ ^t mec^anjLsm. Similarly,^ the career 
developijaent program of the VA is reviewed am^ funded through a 

Separate set of procedures. 
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m* EWBRSy J^ESEftRCH AW) DEVHflPMEMT ADMItflSmTION (ERD?^), DIVISIOW OF 
^ BIOMPICftL AND ENVI!«iian!AL RESEARCH (DBERT ^ 



Bie major portion of^ the Divisicsijs support,, about 80 percent, is for 
the National Laboratories? the rahaining 20 perc«t is for the support 
of extramural projects. In 197^^ this constituted approximately 
20 QiUion dollars. 

saSA siqpportfi solicllied contract proposals subptitted in response to 
a request forHproposal (RFP) as well as ^insolicited j>ropqsals. Hie 
latter are similar to NIH reseacdi grant proposals arel procedures for 
their review will be described. . , 

Orteolicited proposals may be sufcroitted at any time durifig the year. 
Upon receipt, review is begun immediately. If handled normally, the 
review can be conpleted in tlitee months or more rapidly, if necessary.^ 
Unsolicited proposals are referred to the appropriate Branch 
.(currently there are five Branches^J^reviewed by the Branch Qiief , 
assigned to the tedinical staff representative v^ose scientific 
technical expertise best matches the res^rch proposal. 




13BER use's a mail review procedure. A list of reviewers and their 
specialty areas is maintained and a g^ieral effort is m^e not to 
4jse the same reviewer too frequently. The tedmical representative 
sends the proposal to 3^6 outside revieweraj^and receives written 
•reports from each** The consultants review for .sci«itif ic merit only 
%^reas the staff representative reviews for relevance to the mission 
and fhr budget considerations. The staff representative hac . th e 
authority to reject the recomnendations of reviewersT but if he does 
so, he^must defend his action. 

The staff r^resentative's suitinary and the proposal are circulated to 
relevant tedinical staff within the Division for ir^wt from .other 
'experts, usually so-called "category teams" from each of several 
Branches of the.Divisicm who review the proposal. The proposal and 
suninary report are thoi reviewed by the Research Committee of the 
Division, ^ich is coipoeed of the teciinical" staff of the Division. 
This Connittee meets weekly and the responsible staff representative 
presaits a summary, reconroendatioDr and the reviewers* opinions^ 
eadh application. The Research Committee votes approval or disapproval 
and japi^oved proposals are ranked for payment. 

There is one budget for the Division. This allows for flexibility^ 

to shift funds where needed to support approved projects. If necessary, 

approved projects can be held until funds are available. ' • ^ 

In 1975, the. Division supported «4x>ut 5(Jt) contracts and there are .^bout 
400 renewals yearly. Ongoing ijontracts are r^iewed by ouftside con- 
sultants every third, sixth, and ninth year and the reviews are doqa- 
ment^. New proposals over a specified dollar amount imist be ''site 
visited," usually by the staff representative, who must submit a written 
report prior to the initiation of the r^ular review. 
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IV. HftTIOtiAL mtyPimCMj AND SPACE A3ENCY (NASA) , OFFICE OF LIFE 
SCIQOS (OLS) ^^T" ^" 



The NASA program is undergoing restruc±uring«£rom a rolssion*-oriented , 
program to a problenh-oriente()£program. Decentralization of funds and 
problem clusters ar^ to be allocated to the three field laboratories. 
The procedure prior to the planned restructuring was" to hay^Nall 
prp^osals received in ttie c«itral office, then sent to the relevant 
Celd laboratory for prescreening prior to initiation of peer review 
lich was cOTduCted by the centtal office. Ai)rief discutelon bf 
general teview procedures will follow.' 
* * '* 

NASA carries out its mission via contracts and grants: contracts 
with industry, grants with universities and non-profit institutions. * 
The nature of tb^^toject determine the choice of instruitent. Hie 
review of unsol\cited grant proposals will be describe^l briefly as 
these proposals are similar to NIfl invej^igator-initiated grant 
' proposals. ^ - J ^ 

Proposals ate reviewed for scientific merit by panels convened 
quarterly. A contract with^ the American Institute of Bi61ogicaX Sciences 
(AIBS) provides for the manageroait of panels, ,selection-of members, 
conduct of review, and pre^ration of written reports evaluating^the 
proposals. Mem^rship on panels is on an ad hoc basis. ^ Selection of' 
reviewers is made as required to provide the expertise for the 
proposals to be reviewed at the specific quarterly meeting, niere are 
deadlines for receipt of applications to meet the quarterly panel 
reviews. Although AIB& selects the panel m^rberp, there is input to 
thV selection process form OIS. Ihe panels review for scientific ^ . 
merit only, reccnmeraing approval or dise^roval* If approval is 
^^iCI^iiiiended, the proposal is ranked for quality from 1 to 4. 

Intermediate rankings for quality often require additicml information 
and/or negotiations with the applicant. OLS gtaff reviews proposals 
for program relevance. OLS staff preparjBg""si3r!inaries and recormendations 
based on the two reviews. Funding decisions, made l^y OLS staff, are 
bas^ on the scientific recqnn^atic^ns and^mission relevance. 

In 1975, it was estimated that approximately 3.5 million doll/ars was _ 
. available for researdi grant support in biomedical research by this 
agency. - ' -r ^ ^ 




OFFICE OF NAVAL RESEARCH (OHR) . 

• ■ . " ' i» • 

GNR siQjports m^ssion-^riented basic research in four program areas. 
. Proposals are reviewjed ooi^ide^ consultants convened on an ad hoc 
basis. ONR contracts with AIBS to select consultants and to concluct' 
the panel reviews. ONR has no statutory requirement to provide peer 
review by outside consultants, but finds such reviews acc^table, 
and is generally guided by the recommendations. ONR may and * 
frequwitly does reconin«rf consultants to AIBS for the panels. Bie 
consultants review for sci^itif ic merit only. Proposals are rated 
on a rating sheet designed by the panel. With the information 
provided by the reviewers, ONR slaff decides v*idt should be supported, 
ranking proposals on the basis of relevance to the roissicm of the 
Navy. / / 

Most contracts are for 4-5 years and there is about a 20-25% 
turnover. In 1975, ONR supported about 300 contracts including 
renewals, totaling approximately 15 million dollars. ONR does not 
anticipate an increase in funding or expansion of this program. 
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VI . ' UNITED STMES DEPARIMENT OF ASRICUL'TORE (USDA)? €OOPERATIVE 
STOTE RESEARCM- SERVICE (CSRS) ' ' ^ 



CSRS awards grants and a small number of cooperative agrfeements. 
Additior^ally, it has two formula grant programs ^md a discretionary' ' 
^peci^l grant program. Hie special grant program is mo?t comparable 
to the NIH extramur^al programs. In 1975, CSRS furxSed :abput 150-160 
projects for a total of ahcxit 15 million dollars. CSRS anticipates 
about the dame level for 1976 but may have a slightly higher budget 
approximating $19.5 million. 

CSRS staff obtain outside expert opinions from "evaluators" for •each 
proposal. Selection of eScperts is made by staff . The nianber of * 
evaluators may vary according to the proposals. A standard list of 
ten technical and two_"inpact* criteria is used, for evaluating 
proposals and a 0-10 ranking syst^ is used. CcHiflicting recornnenda^ ^ 
tions of evaluatdrs are resoJveCby further discussions with CSRS 
^ . staff. Additicmal information may be^ Solicited from the applicant, 

and additiOTad evaluative opinions may be sought. Projects to be 
supported must fit program allocations as identified by the P^^rtment 
or by /Congress. • ^ 

Budget allocatiais in various program areas, criteria for review, and 
format for grant proposals are published in the Federal Register . 
Proposals are. received within one month after announcemeat in . the 

^' Federal Register and the review proqess proceeds immediately there- 

f after. . 



Rejected applicants are usually "provided with ^summary of the critical 
elements vrfiich may have accounted for the rejectic^. 

Bie program jof the Agricultural ^Research S^vice, USDA, was supported 
in 1975 by an allocation of c^roximately $250 million of vrtiich only 
$2 to 3 milliOT was available 'for funding extramural projects. Grants 
are used to support and supplenent the in-house mission, aftd onlj; if 
by so doing, it will be more efficient and less costly than conducting 
the research in-house. No detailed review of this type of granting 
was made as this appeared to be research directed by in-house staff 
and more nearly approximated NiH contractual research. 
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The JHIH Grants Peer Iteview Study Team' has not had an opportunity to study foreign 
systems for the review of r;esear<* prqposalp in a careful and systeroatifc 
manner. ^ However , many individuals on the NTH sp^f ham had firsthand 
experience or coiitact with foreign review systems. It is significant that 
there is not available an up-to-date and coipr^ensive colledfelcm of. , 
information concerning the'se foreign systems. In our judgment, sudi an 
lexdiange of information among the research administrators who are primarily 
responsibly^ for these functions would be useful; information and documwi^ 
tation concerning review philosojAiies and procedures should be distrilDuted 
widely* Ihis mi^t be an appropriate activity for the Pogarty International 
Caiter to sponsor. ' 

/•* ' jf ' 

Ihe-roertoers of the NIH Grants Peer Review Study Team have had available 9 1974 
T^rt prepared by Dr. Kevin O'Brien of the Nation^ Health and Medical 
Researdi Council«f Australia. The Australian Governmeit h^ increased 
the amount of fui^s made available to the Council at that time and it was • 
considered opportune for a study to be made of medical researdi adminis- 
tratiOT in other countries. Dr. O'Brien, under a Wdrld Health Organization 
fellowship, visited those institutions "most experienced in research 
actaiinistpation in North America, the United Kingdom and- Europe,^,*' His 
156-page report entitled "The Organization and Administration of Medical . 
Research" served as .an important resource. (20) The Study Tfeam is not 
aware of any other systematic study of peer review policy and procedures 
in various countries; ^ch surveys would provide useful information. 

* *' " * 

-In" digcussing the iasic pri|Bp?les of central researdt^rganizations. 
Dr. Q'Brien connented that^R^was clear from my, study that no perfect 
system has yet been developed for reviewiaig researdi grants— although 
the NIH system j.s about as just and as accurate as any procedure can be 
which depends in^the last analysis on human judgmoit." ' ^ 

Furthermore, in describing the peer review system at NIH, th« report states 
that "Ohe procedures followed in reviewing what arte Known as traditic»«l 
research grants were instituted first and caistitute the most thoroughly 
establ idled, the most rigtfffouil^ followed, the easiest to understand, 
probably the most widelW actaired of the activities included in the. external 
programs of the National Institutes of Health... ihe procedures worked 
out.... constitute a' ver^ significant administrative invention..." 

It is alsojjotewOTthy tttA this foreign biomedicaJ. scientist felt that 
"it may be well to point out that years of experience have given the 
scientific coranunity in the United- States a respect for the ability of the 
, staff of the Institutes v*iich it may not have had in the beginning, niis 
re8?)ectand trust provides a fitm foundation for giving a greater degree 
of re^)onsibility to experienced staff mgmbers in the early stagesjof 
foniiulation and review of conplex propos^as in the new program areas." - 
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CANftEA * r 

_ . ^ . _ % V ' . ' 

Since 1969 r when the Governmmt Organizaticxi Act was passec^c the (Ie4ical 
Re^rdi Council of jCanada has been a separate d^rtinent reporting to ^ ^ 
Parliament through the Minist^ of National Health axid Welfare. Itie i 
Medlca]^ Research Council , however, operates outside the framework of the 
Departmait of- National Health and Welfare/' Ihe Council differs from NIH 
and the Medical Research Council 'in the United Kingdcvtt.in that the Councii . 
does have laboratories^ of its' own and does not conduct reseapdi ' ^ 
dir«5tly. X . ■ ;> 

In C^iadaf^oluntary agenciesr provide approximately 20 percent of tlje 3fant 
funds that sij|port bianedical researdi; a modest amount of fundd is available 
from Provincial Governments; the central ^Federal Governroent provide? the 
major source of support. / " * 



e evaluation and review of researdi proposals, as conducted by the 
Medical Research Council, are very similar to the NIH> referral and initial - 
•review group systan. Heavy reliance is placed upcxi assessmait by pe^ 
reviewers. Applications subntitted to the Council are referred to one, 
and sometimes two, of its standing committees. Coninittee iQpi>ers vriio 
serve without remineration are drawn from medical sdioq^ faculties. ^ 
Prequent3.y, proposals are referred to outside review^s who are not mentera^^ 
of one of the Council carmittees. Each plication, alc«g with the outside'^ 
opinions, is discussed in detail by the coAmittee. The assessment includes 
scientific merit, significance of the proposed researdi, ccmpetenqe and 
background of the investigator, previous publications, and! the ^^propri^te^ 
ness of the requested budget. Numerical ratings are ag^igned. K^The evalua- '- 
tions and' ratings of all applications are reviewed by an€xe6\>tive Conmittee 
and then forwarded to the Council for final acticxi. Some particular aspects 
of these researdi grant awards are of special interest) und^r thes^ awards 
principal investigators may not receive aay remuneration and no fundls are* 
provided for overhead^ . ^ ^ 

Hie staff of the Medical Researdi Council, the Secretariat,; is .responsible 
for the actual conduct bf the review and approval procedures. Each y^dv^ - 
there are three competitions for project grant support. A^roxinjately 8d0«=-- 
applications may be received for each of these cycles. Senior staff of 
the Medical Research Council Secretatiat, along with scientific ccxisultants, 
assign each application to the mo^rfTapprbpriate of the seventeen grants 
coninittees or, as ndted previously, tampre than one committee.' Ttie 
.assignment of applications is -^a^ii^wfimarily "by title" of the project and 
taked intQ account the compet^ce of individual committee meifbers. In 
addition to assisting in the.as?i^nment of applications, the^ consultants 
dl90 suggest outside reviewers to v^om each ^ppllcaticm could be s'^nt for , 
conment. Shortly thereafter, each committee chairman receives a;.list of 
applications assigned to^his committee and a lisr of suggested reviewers. 
Ihe conmittee chairman may suggest reassignment to a different committee - 
or dianges in outside reviewers. * The applications are then sent; ouf* for 
review. The committee diairman is alsd ressponsible for the assignramt of 
each application- to two or more members of his committee; they will abnduct 



. « int«idivtt- review of. J^e propdsal anS will initiate the discussions at 

tile ocBBiiet^ Wtin^Vj Each eonAittee ipember/may receive betw^een ten 
I' ariET'^twenty such applieatitsna f<jAiJi-d^th preview. Although each mepber 
of tiie coraraiftee does Vx^t^ receive S compitete/fcopy of every. application, 
. eMh wadber does «{»5Bivejan abbreviated veraricSa of every application 
assi^^ tQjiis.jxmmittjjb. " / • ^ * ' 

plication bqgiks are distributed' tcr comnittee members approximately four 
liiedcs.before'>thiair mSfetings in' Ottawa^, Each comnittee meeting lasts about 
V two. days but th^ depends upon the number of applications to be considered . 
,fl|)plicAions^ar^ /discussed individually anc^ in-dd^th . Each reviewer Is 

• • ceq^ired'to assign a numer'iced* rating to every application; after the 

Seeting these ratings^are averaged" and listed in ordej: of scientific merit 
ji^(x presientation t<i the Gouncil. , - - ^ 

It^ Cduncil^ rarely alterfe .the recommendations 6f the Grants Ceranittesps . ^ 
the Council does\t^J^ into'accoun^budget allocations, the abrupt with- • 
'^ drawal" of reseauch ^jipport, and ^^^Kactors. Budget allocations al^ 

* made by support Biedianisms' (felloiJB^, grants'; develsswhtal grants.) 
.several months'b^pW eadi- ccmpfetition, /rem time to time spe^al projects 

* are^identified as priority area^, e.g. ,, prenatal diagnosis of c|inetic 
tdisieases, therapeutic trial of - human gfgwth hormone : - 

. 1. In receit year^^e Goup<dl, She- Secretariate, and the biomedica]: researdi 
^ \ conpiunity' have been rsl^artiining t^iese proced^^^ Some have alleged' that 
|tt one granQt corimi'ttee Igmuch inore critical todTigorous than others'. 

Connitt^fejBfe^rs \*o ha\)e b^ transfecredl^ran o^ie committee 'to an^er 
. have "-apparently not noted I such differences. At,, times the seme "application 
^ has' been. dellbKateay assigned to -more than x)ne~ Grants Ccninittde— tJie 
differences ih^tingg. and. reeatinended budgets ?»"re ^id to h«ye been". 
minimal. Studies have been jSerforiDed to test thtr efif ect of eliminating ^ 
the highest. and lowest ratings of committee mertbers. Thus, r^jeated - 
^ , efforts have been 5iade to^est the 'objectivity and the <3iscrimination of- 
■^•-the Canadian peer "review ^tem. Concerns continue to be expressed how-: 
^ *ver , .esffecially' as funds ar^ not availablei.for t^suf^port of all • 
' aaBeritorlous researd>. ' ■ " • . ■ 

. ^ UNl W- KINGDOM . , ■ ^ " , ' * 

» ' • . 6 • \ " " 

- » In the United- King(^(?m; medical research' is 'supported in several ways* The 
primary roates involve the universities' and medical .sdiools, the health >^ 
departments* and the Medical Resecrrch Council. The Council has had a 
ial tole. in supporting promisi%-. research projedts and in fostering 
eration- among t;he universities* th^ biomedical professions, and the - 
^errntent.' The impact of position jj^r^ such as the Rothsciiild TReport 
is not entirely clear but^ increased "targeting" and mis§i<»i orientatiw 
seem, apparent. • , 




The Medical RSbefirch Council has^ variety of functions and has estab- 
lished- several advisory boards" an* committee^ Iteview procedures are 



\ * administered by a SecretadlaJ: in the Medical Researdi Council head- , 
quarters. A Heilth scientist administrator (Scientific if^inistrative 
Officer) and a coinnittee secretary are assigned to eadi^iirants Committee. * 
Ihe main support device is the project grant. TJiese grants finance *. ^ ^ 
researdi on a well-defined project that c^ usually be completed within a 
three-year period. Other special typesf df grant^ for equipm^t, travel; 
' *^c; alee also available. Most aq^^lications are received from medical 
schools, imiversities/ and hospitals. Upon receipt, each application Is. 
reviewed by the health scientist administrator to Tae certain that the 
agplicaticMl is appropciate. Each 5qppli©ation.is then submtt\3§d J|paie or. , . 
four Grants Committees xSepending upofi tie coatent of the proposalrliflllie " * 
Committee ih rei^>onsible for assessment of scientific merit, appropriate- 
ness of the budget, ijmportance of the qpestion^ being invest ijfated,) etc. 

Ihe Seer etafiat, staff assigns plications to individual members who ' 
9 reyjLew tne proposal in-dep&i and lead the discussic»i. Each committiee ^ 

^member, however,^' is asked to rate each plication. Often (pinipns are- ^ 
^^ob^ined, from outside consultants aiiS referees, jand theHe are jnade avaija- 
^^^hle tp coni!!itt0e- members. EachNcotinittee moiiDer votes Apriority score 
" ahdime^e ar^ subsequently averaged. ' \ 

iiye sci«itific evpluation performed, by each of the f our Gralktis Committees 
^SU^s Ipurport;;^ tojM^^ind^ndept of the overall^ availability^of funds. * 
^ ^Ihus, a delibet^te Btt^rpt is made to^fltainr uniformity and. coisistency 
based primarily upo^ considerations of scientific merit and ojiportunity. 
Applications that score very high are awarded wiithout referral to a second 
body/ a Research Board — unless they are unusually expensive. Similarly, 
proposal^ that are judged to be quite po6r are declined, iiflned lately. 
Thus, in these circumstances, the Grants Cotmitt^e has th^ final say in 
approval or disapproval of applications but in most cirdM&tances applicar ' 
. tions are further reviewed by a Research Board. 

* ' ^ ^* . * - ' 

J^lica^ions reviewed by the committer are interdigitatad al^ listed ' , 
in priority oi^er and referred to one of two Research Boards. Iljey Biological 
Researdi Board and the Clinical Res^ch Board have responsibility for. the|l|^« 
allocaticSr of awards amcxig applicaticxis and. they takeJ.rtt;o account^ factor^^ 
such as cost effectiveness, other sources of suppoj;t, recurrent expenses,^ 
and program r^evance. " ; . • ^ - 

- > * 

Ihe Swedish Medical Researdi Council was organi^sed in 1945 4L the govern- ^ 
mditar agency primarily responsible for the suppoj^t of bicm^cal researdi. 
OjB^ qgQtion of Council deals with general meaicine while another, com-^ 
'^^Bed chief ly 'o#acroed forces representatives, d^als with military medicine. 

^ . Bie major instrum^it for research support is the project grant. Proposals 
^ ar^MBubmltted by Individual, scientists and these are reviewed for scient;^c 
and tedmical merit by|discipline-oriented cocnnittees* There are eleven 
such ccmltaittees and each ccrmiittee has approximately^five menters. In 




certain areas, deemed to be of high program relevance, special groups or 
committees are established,. e.g., alcciiol ism,, drugs, and occupational 
medicine. ' • . ^ . ^ 

%^ BcS^ Of the activities of the Swedish Medical B^sardi Council is a 
reflection of the country's soE*iisticated'medica^ research status. . 
^^5«:oximately fifteen researda units a^e supported by the Council ; in these 
units both* tenured 'and non-te^^ffe3 faculty appointments are suft)orted. 
The CourK:il also supp6l^ fellowships in basic and clinical resev<di and 
WMdi «i{*iasis has'b^en givi^n to bf Edging these two areas of investigation. 
A rither substantial portion of Medical Research Council support: is allocated 
to clinical investigation. ^ • / - 

Although not 13 irectly related to peer Eeview activities, it is of interest^ 
to, note that the Council supports planning activities "Through the use of 
extramural consultants as wd.Vas inforrnaticwi activities (bivalent to 
MEDLARS in the ii.S.), animal ^resources, and' stu^Hes leadi-ng to new/modal- 
itiesrin prognosis and therapy. ^ ' ^ ^ 

awITZERLftND • , • . 

•The federal agency in Switzerland that supports biomedical researdi is the 
^iss Natioud. Science Foundation. Actually t^is is a private foundation 
but it rece|.ves the major portion, of its funds from the government. Aside 
UpBLtlie tjiolggical artd'medical sciences, the Foundation alsd covers the 
"humahities and physicsf '^eBe.is '^Council for each are|^ 

Within^e Foundation, the^iological'and Medical Sciences Council has 
twelve members, most-bf whom are university professors j , The members, 
4er^ four-year, rotating terms; the Council meets frequently— ten to 
twfelve times per year. .Most of the*support of medical researdi is 
dikindled through units that are attached to institutions. ^As at NIH, 
applications for support are reviewed by a review gtoup system. In fact,^ 
the Council ^serves' this role. Site Visits ma^jfcbe conducted by, members 
of the Ccjancil and it appears to be almost routine tp request evaluations 
;from (Xits*ide( reviewers, . Although 'the size -ot, the country makes the 
• performance of site visits quite^ reasonable r bther problens do arise; 
the availability of reviewers is limitecjl and" such* reN^i^wers are often 
cx>mpeting for gra&it funds at the sarnie t^me. 

Trtie F6un^tion has two deadlines each year for the submisison of appli- 
cations y^^pprox^Shately 500 applications ^are. receive in the biomedical 
area each vear! Each application is rated and then listed in •'priorif 
order" bjf^e Council. Final decisions on gj:ant applicaticMis are made 
i>y^^Kg^wiss National Science Foundation sitting as a conwittee pf tl^e 
whole. In general projects are not funded for more than three years . 
at ^time. Grants may ptovide for salaries; equipmait, and consumable 
.43uppiies.* VirtuaUy all of the gt»itees^arp employees of universities. 
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A relatively roodeSt. amount of funds is reserved to support research 
professorships at the universities and a antalJ^ budget is also provided 
for.pre^^ and postdoctoral researcA f ejjpwships. - * 




00t<CL0SI0N ^ - 

UuiSr it can be seen that roost other U. S. Government agencies and many 
of the wstern countries have.scme port of a review syst^ ^ich utilizes 
persons with specific expertise as reviewers although nor^ of the systems 
is identical to the dual review syst«n used by NIH. ' • 



STODIES OF PEERi®(IEW SYSTEMS 
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1. Itie Nation's Medical Research. V^rote of: Science Public 
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2. Olie Advancement of Medical 'ReseatchJInd Eflucation 
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» ■ ^ • . _ . ^•■.i' ' 
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v- ^ ' • ' V 
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CURRafT PEER REVIBW SYSTEM - 
" i)ESCRIPn^ 

In fulfilling its mission, one of the functions'of the 'National Institutes 
of Health is to provide for the control of human disease through the 
support of bioroedical research projects select"te for scientific merit 
and -relevance to health problems. Because of the magnitude y diversity, 
and conplexity of the NIH resear<ii mission and the desire to obtain 
the best advice possible, NIH draws on .the national pool of scientists . 
actually engaged in research to assist and participate with NIHj in 
the-selectiOT of the best research projects to implement a bioeptedical 
research program of the highest quality. 

Hie NIH peer review system is based upon two sequential levels of ^ 
review, referred to as the;"dudiL review system." Ihe first level 

• involves panels of experts, generally established 'along lines of , 
scientific disciplines, such as biochemistry, pharmacology, physiology 
etc. Itie scientific ?^ls, called Initial Review Groups (IRGs). 
consist of fifteen to twenty members and have, as th^ir primary func- - 

' tion, 'the review and evaluation for scientific merit of the research 
grant applications, submitted to NIH for consideration of support. 
The NIH teceives annually approitimately 15,000 applications requ^t 
support, whidi must be reviewed by these panels. . - ^ ' 



r 



The second level of review is carried out by tlje statutorily mandated. 
'National Advisory CounciVBoards 1/ of the Bureaus, Institutes, and 
Divisions (BIDs) of NIH which award grants. It is the r^ponsibility 
of the Council to review the appropriateness of the technical merit • 
review recannendations made by" the Initital" Review Groups arid to make 
a final review- for scientific merit. The Council must also iftake 
recannendations as to program relevance^and priority.. With only very 
rare exceptions, the BID cannot make an award" unless the Council • 
recottmends an application for approval. On the other hand, a BID is 
not required to award a.gr^t singly because an application receives 
a favorable Council reccriiendation. The Council also has the broader 
function of advising on the, overall program of the particular BID as ? 
well as the legal responsibility for reccmnending which projects are 
be awarded. ^ . , 

- Ihe dual review system permits separation of the assessmait of pro-," 

- jects for scientific, and technical merit -from subsequent policy ; \ 
' decisions concerning pcograninatic, scientific areas ih which projects 

will be supported and the level of resources to be allocated to those 
• aceas." By such a s^ration, a more objective evaluation is pbt^ned ■. 
than would result from a single level of review. The dual review^ 
system,' because of its advisory character, provides the responsible 

' * - - ' 

1/ The ^ional Cancer Inst itute and tt^e National I<ibrary of Medicine 
. have Advisory Boards. All of thes^ advisory bodies will be 
referred to as CourK:ils. 



federal officials with the best advice possible in regard ^to both 
scientific and societal values and needs, A brief description of the 
procedures involved inJUH peer review follows.* / ^ 

Rfeseardi^ant and all^ other grant applications prepared by invest!-, 
gators are submitted- through their ins^ituticms* to the Division of. 
Iteseardi Grants (DRGj^ which serves as the central receipt point for ^ 
all of |he applicaticms of the BIDs, The WG reviews eadi s^jplxcatiOT 
for relevance to the overall NIH mission and, if acceptable, assigns 
it to the.^rc^riate IRG for scientific merit review and to the 
appropriate BID for ccHisidepation of an award. 

The IBSs meet three tiroes yearly to review the applications: assi^ned-"^ 
t^ theni. iHGs are composed primarily of nonfederal scientists 'selected 
by NK^or their competence in the particular scientific are® 
necessaty f or the critical review of educations, in advance of the 
meej:ing> the- Executive Secretary of the IRS studies all of the appli- - 
cations assigned to his/her group and c^Jtains any additional information 
deemed necessary for the revi^ from the principal investigator or 
appUcant-instituticxi. Each plication is assigned to two or more 
menrers of the IPG, each of vrfiom prepares, in a<5yance of the meeting, 
3 detailed^ critique of the^ applications assigned' and who lead the 
discussion of these applications at the IRG meeting, -Six to eighty 
weeks in advance of the meeting, every meiriber of an IBS receives a 
oon^lete set of the applicatiorfe sdieduled for review ^t that particular 
meeting, in addition to the applications for vrfiich they have been 
designated as discussion leaders, the men4)ers are asked to become 
familiar with the remaining applications* During the six to eight 
week period prior to the meeting, the members of the IRG often ccmtact 
the Executive Secretdfry for additional information, if needed, for 
the review of a particular application. In certain instances, this 
information can be obtained only by visiting the ^ Investigator {«) 
at his/her institution, Ihe-specifiC details of a project s^e visit 
have been de^rlbed by Merritt and Eaves in their article entitled 
"Site visits for the review of grant a^splications to the National 
Institutes of Health: jview? of an sqpplicant and a scientist adminis- 
trator, " (1) " . . ^ 

Eadi explication is discussed individually at the IRG meeting. It is 
the r/esppnsibility of the IRG to review c^jplicatiais on the basis of 
tedinical merit and scientific significance, whidi includes: 1) an 
asseS9ment of the in^rtance of the prppc»ed research problem; 2> the 
novelty /and originality of the approadi; 3) £he training* experience 
and research competence br_^promise of the investigator (sj^ 4) the 
adequacy -Of the experimental design; 5) the^itabillty of the 
fTOilities; and 6) the appropriateness of the requested baSget 
relative to the work, proposed. Hie recoramendationi of the IRG is made 
by majority* vote of the members. . If recomnnended for approval, the 
applicatio*T is assigned a technical merit priority rating by each ^ 
individual I|^ member, •'l" being ^the roost roeritprious and ^'S" the 
least meritorious. Tlie arithmetio average of these individjaal scores 



isWtiplied by 100 ai* this numerical rating becomes the IHG s raw" 
priority score. An IH3 may also vote to recoranend disapproval of an 
aljpllcation or to defer it for further information^ later reconnen- 
datioh. Following the meeting a suninary statement is prepj^ed for eadj 
aoplicatioirreoontnended for approval or disapproval; the suniharyM pink 
Seet" in the case of DPG initial review groups (study sections)]^ , 
contains the recoranendation, the priority score, if the applicatir- 
been favorably reccranendecnSid-a sunination of the reasons for tha 
recoBinendation. This statement is forwarded to the a)ppropriate JID 

• for the next level of review— caisider at ion by tiiat unit sUatic - 
Advisgry Council. 

In general, the National Advisory Councils also meet three times 
yearly for approximately three days, and each Council member receives/ 
in advance, copies of all suijinary statements releveant to the BID for 
which they are advisory. Copies of the application are^so available 
to the Councari«rbers, if specifically requested. The membership of 
.the National Advisory Councils is set by law. They usuallx^onsist 
of twelve or more menbers, including scroe who are scientists and 
others who are lay coraminity leai^ers intgrested in health areas 
relevant to the' program areas of the particular ^JD. ^ . 

Following the above action? by the various peer ^iew'gorups, the, . 
BID staff makes its program decisions on fundiMf After the inves- 
-tiqator is notified of the recommendation of^e Nat lonal Advisory 
Council, he/she may request the BID. to send a suninary of the corments 
wh*ich led to the action taken on the application. In the case qf 
disapproved applications, ot> approved applications which are not 
awarded, these comnents can be very helpful if the investigatbr ^ 
' wishes to submit a revised application. The procedure to be followed 
in submitting a re\^ised application is described in the instruct tons" 
which are- included in. the research grant applicaton kJ,t. ' 

It is interesting to note that the cost Of this review process, from 
thfe time an'applicatoin is i^eceived to the making of an ^ard, is a 
very smill percentage of the amount of funds awarded.. In fiscal^ 
year 1976, approximately $1.4 billion were awarded for competing and 
honccnfwting applications, including indirect costs. Ihe cost of the 

• peer review prSa^ is approximat;ply $15,800,000. Ihis represents, 
about one t)ercent|of the total funds awarded. ♦ 



A detaUed description of the peer, review system is provided in 
AppKidix C. ' 



CUFJlEWr PEER REVIEW SYSTEM— DESCRIPTION 
References ^ 



1, Merritt, Doris H., and George N. Eaves. Site Visits for the Review 
' of Grant Applicatioi^ to the National Institutes- of Health: Views 
of an Applicant and a Scientist Aininistrator. Fed. Proc. 34: 
131-136 (1975). * 



- 1 



V 



* ■ . ■ ' • ■ 

SEPARATION OF REVIEW AND PROGRAM ETJNCTIONS 
• ' 

-^^he Grants Peer Review ^tudy Team unaniinou^y a^^J^rongly endorses the 
principle of separation of review and program staff f^^^^ions atNIH. 
in some respects it has been in effect at NIH almost since the teg inning _ 
of th^^ctSmural qran^ProgriT T»le -respect and rotation of the Initial 



. of thefixtranural gran^pisMr^^. -respect and f^'^anion or .n...^ 
, -Review 9roup or -study se^tl6n' i&aratus," under the Durisdjction of the 
Division of Research Grants, is a testimonial to this principle. 

. ' The Division ofResearch Grants (DR3) conducts initial scienttf ic and-^technical 
«^it review of regular research .grants* or the 11 NIH institutes, the 
National Library of Medicine, the Division of Research Resources, and 
-s^veSfother segments of ^e PHS. It is a separate 

within NIH, subservient to none of the BIDs it services. Organ izAtionally, 
' !t is ac^table only to the Director, NIH. Implicit in this arrangement 
is the fact thajt the review of regular, research grants nnder^e _ 
management/ of separate I»G professional staff wfrich has no responsibility. 
', forlthe mahagemSit. of grant progtairS; FurxJing decisions are made.. at the - 
program level with" ifT the BIDs. 

There are nhilosophical and practical reasons for this arrangement. The . 
Ir^a^^t prSSs a buffe? to prevent emotional or political considerations 
t^^Sn^^io the decision^king process. Centralized peer review 
offers th6 advantage of greater efficiency in utilizing consul tantsto 
review^the very large nunfcer of applications received each year. Unnece^ary 
duplication of consultant manpower is avoided because arelications^an be 
clustered into circumscribed areas of science. Moreover, having the 
review apparatus centralized in one organizational locus allows for 
Ji^lSriS^. surveillance ancT fosters the development of uniform review 
' practices and 'procedures. , ' . . ' ' • 

Ve NIH has a wide v^iety of grant activities. Whereas the regulaf 
Research grants, fellowships, and the research career development award 
applications are reviewed b^ the DRG study sections,, applications for 
^ nSv of the other grant' programs such^ as .program projects and centers 

"^rSiewed *^e BIDs. Over the^p^t few years, there has been a trend 
^i^hirS Bl^Tro emulate the DR3 system. Thus, the ^^^ional Heart Lung, 
^.Blood Ijistitute has created its own peer review system to evaluate the 
Cny special grant applications. Other* institutes attempting to do 
the s^ For exairple, the "National Cancer Institute has a centta^ i^ed 
peer review system for ti^review of program project and center grant 
1 aoDlications This pattern is gaining momentum at NIH because of the 
VJSiS^ Te^tai^ prSrams and 'the recognition that the DRG model offers . 
iS^ldJantages. Th?DRG model is especially attractive sincS the reviews, 
being conducted in the institutes involve support mechanisms which have , 
te^sSbject to some criticism in the past, incl^uding l&rge gran^ which ^ 
are' complex, involve large ^unopnts of money, and must be subjected to 
th^S^^r review to matSi that carried out by the study sections. 

'*rtie term "rwular resear ch grants'* ts used to designate investigator v 
initiated rSearch grants coded as ROls, aS distinguished from program 
project, center, or other special, multipurpose type of grants. 
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Because of the very special nature of these Institute initiated programs, 
it is unportant that there be good coordination, good planning, and 
good communication among Institute staff to accorplish effective, 
critical, and reliable review. It is important that staff 'carrying out ' 
.reviews understand the objectives and the'gpals, ;iot only of the Institute but 
of the particular initiative under', consideration; -they nust also be thoroughly 
conversant with review policies and procedures. This -review responsiblity > 
should be placed in the hands of a st;aff that cantapply uniform standards of • 
peer review for the support mechanisms under its i^isdictibn. At the same ' 
time, t^ r^iew .staff must be privy to program p^s as they develop and 
mature fh ordet to fully grasp the intent and background of the initiatives. 
It is equally important that program staff Tesponsible for developing the- ^ ' 
initiatives and managing--the awards and contracts after they are made, be * 
given the opportunity to explain such Initiatives to review camittees. What 
emerges -from this brief ^wrtrayal is a dichotomy of profesSic«al staff ' ' ' 
functions: the function related to review and the function related to 
program manageroait. As in the case of the I»G-BID relationshit for the 
regular research grants program, the professional staff functibns within a 
BID should be^kept distinct and separate. ( 

In addition to scRie of the reasons already cited, one of the mkjor advantages 
of this s^ration is to avoid prejudice and partiality or circumstances in 
•vrtiich there may appear to be undue emotional or intellectual cdnmitroent to a 
particular approach. For example, even the most well-intentionW and virtuous 
program manager is suspect v*ien he authors a Request for Application, selects 
the consultants *to review the responses, chairs the review nifeeting, tallies 
the votes, writes the summary statements, selects the applications to be 
awarded, and serves as the orogram administrator; The Study Te4m believes 
that aubh Situations must 5e prevented; the credibility of award decisions 
must be carefully guarded. To this end, the establishmdit of a/peer review 
system within an Institute, in the hands of a professional staff detached 
from financial involveifent in program develc^ment and management, offers the 
best preventive r^nedy to insure the continued well-being and integrity ot 
initial peer review; In order „to maintain confidence in the system and to' 
assure adherent to the principle of separation of review and program staff 
-functions, continued overview is necessary at both the BID and NIH levels w 

The separation of review and program staff responsibilities has been 
demwistrated to be not only wprkable but wise, it has increased the 
objectivity and thej:redibility of the entire peer review system and 
this, in turn, accdCnts for its durability and the confidence in the 
system that, presently exists. 



THE K3LE OF. PEER REVIEW IN DECISIOti-MftkiNG BY NIH 

Although the primary focus of its efforts.has been on the assessment- of 
Sientific and technical merit, the Study Team believes it is useful to 

■ reflect upon the role of peer review in the broadfej: contexts of decision- 
making at NIH. Since the very inception 6f extrairural programs at NIH 
peer s%view processes have often played a central role in many phases of 

■ decision-making.' These processes have been utilized in program reviews, 
in determihing program relevance and priorities, "in setting or resetting 
program balance, and in the planning and evaluation of programs. .:. • 

The Initial Heview Groups (IRSs) take part in one of the first and probably 
the most insxjrtant ste?s of the aforementioned activities: the identification . 
of projects that are meritorious and worthy of suFpgrt, using scientific 
merit as a primary basis for judgment. 

. The -Institute Advisory Councils and Boards, play a'major role in considera^ 
tions having to do with program relevance and program balance. They give 
advice ar¥3 guidance in the identification of projects which will provide 

: for a balanced program and which will bear significant, relevance to the 
program objectives and missions of the specific Institutes. In making 
their judgments and recontnert3ations,. the Councils are mindful of the funds 
avail*le to the Institutes^ and the reconmendatidns they niake are based 
on their best judgment for the appropriate use. of available funds. ^ 

on ^continuing or intermittent basis, individual consultants or groups of 
consultants are assentoled as task forces, advisory comiittees, and comnissions, 
and (us-ing'such mechanisms as workshops, symposia, or regularly planned 
meetings) they provide useful information and suggestions to Institute 
staff on program plannirig and program ev^^uation. THese consultants provide 
advice and' counsel regardTr^ research, progr^ and training areas where such 
information is needed, and they assess the appropriate timing for the 
initiation of programs to translate research knowledge into clinical ^ 
ajjplication. > ' , 

Traditionally, NIH has always relied heavUyon the input received from 1 
its advisory groups. Although program decisions on funding, whether for 
program balance, program relevanee or priority,, or for other considerations 
- are the ultimate responsibility of staff, nevertheless, J±e ^ood oudgment- 
provided by mentoers of peer review.-grouJ>s in their capacities as IRG meirbers. 
Council members, pr ^ hoc consultants has formed a b^s for a close partner- - 
ship and trup^ between consultants and staff. \ 

t ' « ^ • , . 

■Strenuous efforts are m^e to insure that consultants of the highest 

quality and Capability are selected. Yet there is a need to insure geo- . 

graphic and institutional balance as well as female, "minority, and youth 

representation. It is^also recognized that the individuals selected as 

IIG and Council members frequently compete for NIH funds to support their 

own work; this is a natural consequeqfe of a review system that rightly . 
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inportar^ce oiv selecting reviewers of ohtstanding* competence. * ' - 
, Jhe agvAce of jMch. experts is considered ess"enFl»«a^ therefore, steps 
nav^ Been taken to avoid conflicts" of int;erest. 

-Seme of the problems confr-dnting. advisory groups, and f^oh affect thei^ 
" . reconinendations in re\>iew, have to do with their own perceptions of th^ ^ 
appropriate balanpe between targeted and basic reseafdi, the bailee ♦ 
between* funds to Tae allocated fo°r contract versus grant-supported activi- 
. ties, and the balance between projeqt support or other- support of large 
i ^ programs or centers. For exanple,in one Institute, the Council*has the 

specific^ legislatively mandated responsibility -to provide advice 
concern irSg , the per)bentage cJf funds to be used' for contract supported' 
' research. In recient yeaf s , the fidvisory Councils and Boards have become' r ^ 
• very much carteefned with the balance of funds available Titor traditional 

investigator^initiated research projects^r^-that availab^ for the larger- ^• 
^ mechSnisiiB of support, i.e., t)rogram projects, research centers, and other 
- .^>ecial purpose,. irultidisciplinary grants. In some instances th^yllye ^ r-' 
reccnraeDded a ceiling f6r each type" of support. Thus; the impact of 
the various advis^»Ar coermrttees on such d^isions is increasing. The 
Changing character ,b^_jfche times, the limllM availability of funds,' • . ' , 
OTnsideratior^, of -.accountability, the increased participation of public • ' 

^ '. . menbers, andWie increasing |isibirity given to advisory ccninittee 

reccramendations will ffontinul to contrit^te to a changing pattern^of 
* - decision^naking. 

- With all of "these factors arjd influences weighing upcxi NIH decision- 
"^.jnaking, the assessment of quality aiKJ merit must be maintaijied as ttie 
axis iderat ion of jover riding i^rportance. 




PERCEPTIONS OF ^TOE NIH <SRANT& PEER REVIEW PROCESS 

INTERACnOW OP STUqy TEAM WITH THE SCIENTIFIC AND lAY PUBLIC ' 

From ttje inc^tipn of the activies of the 'Grants Peer Review Study 
Teean tGPRST), it, was cleSir that there was a need to obtairithe 
perc^ions of individuals pther than NIH staff about the wiff peer 
review system, ©pinicxis were actively solicited fcoro several' gfroi5>s 
namely, the scientific 'public, the general public, and those v^o 
participateyffT&fie peer review process as members of Initial Review 
Groiflps and Mvjrory Councils., The^ interactions are siitinarized below: 

1. Letted Responses froffn the Scientific Conrotmity and the Public 

A^forroal announcenent of>*he establishment of the NIH-GPRST 
appeared in|the August 8, 1975, issue of the NIH Guide for Grants 
,and Contract^ and in the Septetiber 4, 1975^ issue of the Federal 
Register {Appendix A). Included in the announcement s^«s a 
Btateaent' that the Study Tedm would be pleased to .receive 
written ccmnents, views, and reliant information from the 
scientific and general public concerning ±heir percepticxis 
6f the NIH- peer review system. .Subsequaitly, in order to 
increase the response, a new call was made at the time the 
Study- Tean announced that, it wbuld hold open. public hearings. 
. This new call wa3 announced in the DecQiber 8, 1975> Issue _ 

of th? Federal Register . ' In addition,, a mCTorandum was mailed 
""to. a totid of 30,60O grantee-investigators, disapproved applicant 
investigators, those who^e applicaticxis were 'approved but not 
*funde<?, potential investigators, scientific associatioes, ' . 
/ public interest group&, key officials of interested institutic«TS 
(including minoj^ity and.anall sd^ools) scientific women's 
prganizations, scientific newsletters, and all others on the^ 
* mailing list for-th^'NIH Guide for GrantJfe and Contracts * . 

^Apperjflix B). Within sJeveral months, approximately r,5p0 • 
' . letters were received. ^ The letters came from a wide variety ^ 

ofVindividuals.^ Many respondents identified themselves as 
r „grantees, *as "disappointed" applicants iAc©e requests had 
' .been disapproved pr approved but rfot funaed, or as present or 
fdtmer Ad>^is6ry Council or Initial' Peview Group members. /MoSt 
^fepQke for themselves while sdme indicated that their letters 
represented the views of -a'g^gup or i specific organization. 

Although many iijdividuals strongly endorsed the present l!iIH 
> review system and pleaded that ij: not be papered with", 

many others were critical orvrrtade suggestions for specific 
\, chartges. Ih^ letters expressed the respondents' views ' ' 

on a f broad range of issues, arid a great variety of subjects 
I were discussed, ^ However, review of the. 1,50.0 le[ttfers indicated 

that four is^sues were discussed most frequently: 



!• Hie select icm* of initial review groip members, ^ . 

irt?luding concerns such^^s bias and cronyism^ 

2. Cdncen^ssover'the iirpact o^ the Freedom of Information 
and.PriVao^ Acts on ei6 quality of 6cientif ic revieW. 

3. Itie* qiial'ity ^ ayaU,ability of Nummary, statements^ 

• . . ^ ^ «J . • 

4. Ttie need for a mechanism-to permit ^ppealsi^^in regard to 
the reocjainendations of pe^r review groups, as w^ll as 
assignments of applications ^ Initial Review Groips,. 

a/or Instit^ites, ^ . - ' ^ 




'\ Oth^r is5»es wer^ 'addressed i but to a ^'lesser -extent. ' These ' 
' included the quatity of the; review of large grants, the wprkload 
of^ Initial Review GcoOps, release of priority scores, arid prpblens 
dr^upport for minoifity and women investigatory. ^ 

Other in?x)rtant elements expressed in the letters were suggestions 
for improving pr^roodifying the current NIH review syst^. \Many 
suggestions were* presented!^ in detail, irfdicating concern based on 
knowledge and exfferience and were worthy of- consideration. ' : 

^ ^ ^ - ' ' ' * ' ^ . * \i , 

-In reading the letters, tK& GPRST members were impressed by' the 
'^e^ioys and thoughtful nature of the opinions 'expressed. Tlie' • 
perceptions of vthe Ctudy Team wefe considerably enhanced by the ' ^ \ 
review of the letters i; arv3 its subsequent deliberations were K 
"irifluenced by the* views offered tjherein. 

. From a careful reading of the letters; it became clear to the Study^ ^ 
Team, tio^er^ that many statanents made, in good faith, reflected 
a lack of knowledge of the NPH peer review system as well as of the 
specific steps, involved in .processing a gijant application. * For this 
reason, it became especially* important to prepare a detailed descrip- 
tion pf the entire peer review process as part of the activities 
of the Study Team (Append^jc C). > - , 

So that NIH. may. fully benefit from 'comments contained in the letters, 
a detailed analysis of them is 6?ing prepared by non-NJH consultant- 
contractors; The contents of„ the letters will be indexed by subject 
and then summarized. "Relevant and representative excerpts will be 
quoted- All suggestions for <±anging oPrevising the .current NIH 
peer review system will fl^so be indexajT and sijmmarized. Background 
information on respoodentsNrtwdrTrl^^ helpful in evaluatiTig the - 
comments-their status as ajpproved or "disapproved^" Ar^search grant 
applicants, their experience on NIH peer rev^iew groups, and other 
factors-will be Rpesente^p, if available. This^tudy is currently^ 
process ar)d resul^B a r^ expected to be available in late 1977 1 
: and will be included. in Phase II of the NIH Grants' Pe^r Review Study 
Team. Report. . ' * 



2/ Open^Riblic Hearings for the Scieptif ic and Lay Ccxntftonity ' ^ ; 

. The Study Teiam recognized the ,v5alue*of, tHe, letters in providing 
input from the cxxtmunity but it felt that more direct interaction ' 
was also desirable. Accordingly, it.was '^reed that' open public, 
-^-hearings would be an important and very useful means o£ eliciting 
inf ormatibn^ ^j>d recon]roendations» from concerned scientists and others 
affect^ by the grants peer , review process;, Public hearings were 
held as one day sessions during P^ruary 1976 In governments 
facilities in each of three geOgfafAjic locations!. Itie meetings 
• , were held in Chicago, .Illinois, San-Trancisco, California, ai^ 
Bethesda Maryland. Hie Study Teanfdesigrafted a^^>anel pfiits 
" roariDers to attend' all three hearings, the fwielifts routinely 
asked the speakers questions designed 'to^^licit clarification of 
of, oa^ elaboration upOT,~key point;s but made Hcatteinpt to debate 
issjies with* those testi^ing of to^jorrect cl^arcut ihacAiracies 
/tffthe written and Oral confmeqts^o^f^red. TVtenty to 'thirty formal ^ 
._-I0: minute presentations were heard at each hefaring. However, ^ 
^ additional speakers were heard fTom the floor, and the panel ^ 
rdlbined in session until ^1 coninerfts .were ^received. '''Despite ^ . ^ -'l 

* the limited number of presentations (69), tHe St'udy Team felt 
that much useful informaticxi was gained. ' A l^t of gpddkers, 

^. theit institutiQ|fial affiliations-, and transcripts of the Hearings 

testimony are available in the files* of -the GPRSP*; Copies of the / \ 
transcripts may be obtaiTied by addtessihg the* Freedom of Infbrma- • 
.tion Officer; Office of fhe Director , NIH, Beth^sda; Maryland 20014; • 

The speakers were advised in advance that therGPRST was interested 
i in their views on the following topics:, \ 

\ ' l-r Adequacyot the total review system, e.g., ^ 

' . . ~ , a. .Ihe effectiveness of the ^system in ;serving,and. regporwing 
' ^ ^ ^ to societal -needs and expectatipns for biomedical research 

• " , on'disease-retated problems^ , ^ . ^ • * 
- ^ ~^ b. eff esti^l^ness of the system in assisting in.* maintenance * 

. - of a'^trc^V"tii%fr'.<3ttaU|^i^ 

^ c^ The extent Which the systerf assists in meeting the best"^ 
# * - standards of public accountability for expenditure of 

* public funds., ' _ 

- 2.4 Adequacy of the ini^tial scientific review.' 

• \ 3. Adequacy of the -Council- teview. . • J ' 
* ' 4, Adequacy^of the priority rating -syfet^ • ' ^ 

, ; ^, ' , I ' - . 

' . . ' 5. Impact of^the Pj?ivaqi^*.AQt bfi 1974. . • - / > * - \ 

' -6. Impact of the Freedom of . Informatj^OT ftct, as amended in . " 

7. ' Jn?)act of thei£ederal Advisory Committee Act. ; • 
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8. RecoRinerK3dtions as to how the pr^enj^ Grants Peer R6^ciew ^ 
Syst^ can be iitproved. ^ , 

EaxAi speaker was aSked to present a written sunmary, ,or -a general 
/oatlipe^ statern^t j.n full, in acJvance of the hearing. • • 

'Exaroinaticm^of the test iincmy presented at the hearings indicsated 
that the speakers represented 55 different institutic»i? and 11 
organizations. Soq^ individuals traveled la ccnisiderable distance 
at their own expense to attend the hearing .at whi(^ they were scheduled 
to speaflcT An effort .was made by the Study Team to attain a.geogra^ic ' 
spread of institut'ional r^tesenbation asnong the spealcers./. Generally/ 
howeveiTi the speakers were accepted on a '^f irst cdm^, fir^t served" 
b^iSf as their requests w^re received ia the GPRST office^ ' A special 
effort w^ made, however, to er^ourage presentations from minority 
schools and/, indeed, four such instituticwis were r^resented. Tfie % 

Hiesident of •a women's scientifeic Ofgartiaation responded with the views 
of wonen scientists.* Twcx %gaT off icers presented the views of their 
organizations relating to' legdl aspects of th^ peer' review systCTW 
and other, speakers addressed t^hese issues as well., 

A nunt)ef of Speakers expressed total support *for*he NIH' system, but 
others spoke to 'asptects of the sysjtem adbput which' the^had some con- 
qern.-In general, the speakers who expressed concerns ovpr aspects 
of the peer V review process addressed the same issye§ addressed in the 
letter responses: (&) the process by wt\ich m€Ktexs^ of IPGs and 
AiJvisory Gei^ncils/Boards are 3elected^* including 'related coram^nts 
regarding cronyism and the "buddy" sygt«n, (b) the impift of the 
Freedom of Information and Privacy, A6ts on.the quality ^f scientific 
review, (c) the quality and availability of summary statements, and 
,{d) the need* for a peer review a^j^als medianism.\ Several speakers 
spcke eloquently about the need to, support young Aivestigators, and 
the need to support unorthodox or high risk researdi. A few speakers ^ 
presented interesting suggestions for changing certain aspects of trie 
peer review system|suc|i as the creation of a permanent office at NIH 
*to^lan and con&uct experiments in regard to the peer review process. 

It is iiiportan.t* to r&nCTiber that the hearings wer^ held to obtain 
•the views of those who jJ^titipate in, or are otherwise affected by, the 
peer revie>^ process so that the perceptions of the external comnunity 
could be Reflected in the th'inking'of the Study Team. This ^as also 
the case in the letter responses from the external corrrnunity/s^imilarly 
ft became clear 4rom the hearings that there>are misconceptions an^ 
misunderstandings thopghout the external corihiunity cOTcerning the;^ 
operation of the^NlH peer review system, 2uid itbecanfe doubly clear 
that not dftly Was/^ description! of the process necessary. But that 
it should be .wi^m^disseminated4, 

The testimony prte^led at the hearings is^pr^ently being 'analyzed 
. and vcvaluated in d^ail in a manner similar that for the letter 



. responses, and Will be iffei^ed in Phase tl of this report to 
• ' be completed in late 1977. ^ , . ^ . 

, 3. Survey of Initial Review Group aiy3 Advisory Council/OBoard Meirbers ^ 

:o , T*^e third group of .individuals f ran whcwn conments and opinions were. 
' sought by t^ .Study Team was, made up of those who j^rticipate in the 
peer review process as meotiDers^ of Initial Review Gfoi^s em Advisory 
Councils. 'It>is was accotplished by means of a survey quest ioiffiaire 
designed and developed by a subcommittee of the parent Study Team with 
the aid of a consultant survey research specialist and other non-NIH 
ccffisultants. Tlie purpose of ' the survey was to cA)tain assessments ' 
of the existing grant application' peer review systero and suggestions 
for its^improveroent -^om active participants in the system — the 
current roenbers of ^rant review groups and advisory councils. 
More specirfically it v^s planned^at the survey focus on the ^ 
or^ of the group -members 'in three major areas: 

Valuation of the current NIH grant peer revie\^ system. ^ 

Assessment ot the impact of recent' changes and of potential 
future change^ On the quality ax\d effici^cy of the NIH 
grants, peer r^iew system. \ , ' ^ * ' 

Tticms for improving, the e)listing system. 






Foilr o^»r basic decisions were ffiade regarding the survey: / 

*1. The anonymity of all respondents was, to be preserved -in 
^ order -to obtain vil id answers, 

• • ■ . » , ^- 

' - / * ^2. The respondents would-be encouraged to supply oomnents in 
or^er to expand thej^ responses , or to include concerns 
• • . oroitfeed from the gii^tionnaire, * ^ ^ 

3. The survey would include aU current meinbers of the peer . 

, reviews system ^nd not be limited to a sample oP such meitoers, and 

• ^ . A. , Oni^ one survey instrument would-be developed even thoxjgh not 

* . * all questions would' be eijually 'applicable to each of the 
thi^ee fcyMS of- review groups surveyed. — the Division of^ 
/ Researchfljsrants Initial Review Groups, the BID Initial 

/ * ' Review 'GHoupS/ and the National Advisory Council or Board 

• of each ||lD. . . 

, Analysis and 6v»uation of the Survey of Initial^leview Gf oups and 

* Advisory Counci] ™ 



A detailed report of the statistical' findings of this analysis • 
fi^jpears in, Supplement A. A suninar^ of the. major findings of the 
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survey is attached' K Tab' A). AdditiOTally, a detailed conprfhensive — 
repott of the survey, including analysis aihd evaluation of the fconinents 
secticm, ij^ being prepared and , will be available at «i later date. 

A-total of 1/354 <j3€Stic»inaires was distributed to 12 Advisory ^ 
Councils/Boards*, 51 HG-IBCfe, ¥^-24 Inst||ute IPGs during their 
meetings in Uoverober-February 19^6. The questicmaires were given, 
not only to the regularly appointed roenbers of the peer i^evieW, 
groups, but also to liaison members repres^iting Federal agencies 
and to th9se ad hoc cxxisultants attending those meetings whom the 
Exeicutive Secretary considered to'b^ knowledgeable oxiceming the 
review process.** The overall response rate was exceptionally high, 
94%, attesting to the interest and cooperation of the peer revi^ff 
groip members i ^ ^ . 

In examining and ceramet ing on the survey responses, the differences 
in mission and composition- of the Initial- Review Groups and the * 
Advisory Co\j«x:il/Boards must be kept in mind. It should also be 
remembered that the EffG-IBGs' generally review the traditional 
investigator -initiated' researdi ^ant applicaticxis; tiie Institute, 
IRCs generally review the special purpose type grant €?|)lications-r 
usually the large program project t>r colter grants^ *and other typ^s of 
grants submitted in response to specially announced progi;am needs of^ , 
an Institute Another fact to be borne in mind is that the IPGs are 
'the "producers of a product," namely, a simmary statement ("pink" 
^e^) and a pri6rity score for the applicaticm,. whereas the 
Advisor^^Cuncils/Boafds are the "users of the product."' Such 
differeflRs in function may well lead «>ydifferent appraisals of the 
adequacy of a particular product or NIH procedure. 

Approximately one-^«lf of the 1,274 initial review, group and . ^ 
NationaiL Advisory Councils/Boards mendtjers who completed the 1975-76 
Peer ?^iew Survey qu^tionnairj^ alsa-wrote one or more remarks in 
the Siple space provided for jcarments. A slightly higher proportion — 
51 percent — of National Advisopy Council/Board members than of 
initia^l review, group ippmbecs — 49 percent — made coninents. 

Because it was judged important to utiUze fully the comments and 
ajggestitxis made- by this experier^ed aiSJ knowledgeable group, the 
Airvey Subconmittee of the Grant^Peer Rfview '^tudy .Team^ decided 
to sgonsor an in-depth analys'is. Itiis work is .Currently in its„' , 
^itial staged and will be conpleted in 1977^ However, in order 
to enable the Study Team to benefit from Jthe cdtments during its 
dfeliberaticmS, two major projects wep^^inder; taken: 




j^Menfcers ,6f the Board of Regents of the National Library of Medicine were 
not purveyed. 

♦♦Thirteen percent of the respondents w^re liaisOT members or special 
reviewers. 
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base and the resultant canpviter listings of the full ^ 
text of each codnehtr categorized, by applicable survey 
^question, were circulated. . . ^ 

2. Pour analytical r^»rts cm issues 6£ special interest 
tq^ the Sta^dy Ifeam were corpleted. These rQ)orts are 
presented in Supplern^t B. 

Tab A, as moitioned above, presents the major findings of the 
statistical analysis of the survey. 




■A 



\ TAB A 

MAJOR FINDINGS 



_ 'The current NIH system for the review of research grant applications was 
•overwhelmingly endorsed by those individuals who were members of NIH review 
'or advisory groups -at the time of the survey. More, than one-half characterized 
the system as a whole as *lexcellent*' and over 95 percent rated it ^as either 
"excellent" of "good." Equally high ratings were given the general fairness — 
and lack of tias found in the operation of the system, 

• ^ \ - ■ 

The general satisfaction with the current NIH review system was reflected 
in the lack of enthusiasm expressed for recent changes and for potential ^ 
f utur^ ctratTges in the* system arising from Congressional and public concern^ 
with the openness of Government processes. Of 16 such changes, only four were 
perceived by more than half of the review group jnembers as having either a 
"very favorable" or a "favoratfTe" effect on the d'^ality and efficiency of 
the NIH grant review system. 

Even less enthusiasm was shown for the 20 suggested improvements listed 
in the questionnaire. One suggest j^--recommending a study on how to reduce the 
time lajg between' applicatioo and final NIH action — received the endorsement 
of three-fourths of all revi^w'group members. None of the. other l5 suggested 
changes received the support of a majority. 

— • . \ 

The ciembers of th^ DUG Study Sections were generally the most favorably - 
disposed of the three types of review groups toward the current 'NIH review 
system and. the members of the National Advisory Councils/Boands were generally 
th.e least favorably inclined. Even s'o, about 90 percent of the National 
Advisory Council/Board members designated the overall NIH review system and 
its lack of bia$ as, excellent or good. The greater degree of satisfaction of 
DRG Study - Sect ion members was also evidenced by the fact that they were less 
fayorable than both the ^ther two review groups in their responses to 14 of 
the 20 suggested improvements listed in-the questionnaire. ~ % 

. * 1^ * 

• ♦ The survey findings showed that members of the three types of review 
groups varied in certain "characteristics and that some of these differences 
reflected differences in functions among the groups. On the average, DRG 
Stucfy Section members had the 'least - prior experience with the NIH'peer review 
system^ had the greatest proportion engaged currently in research activities or 
as princi'pal investigators .on NIH research grahts,,had the smallest proportion 
in clinical activities, were the youngest, and had the smallest percentage of ^ 
women members. Members of the National Advisory Councils/Boards differed the 
most from DRG Study Section members in many characteristics while Institute IRG 
members frequently fell between the two other groups. In some instances,^ the 
characteristics of Institute IRG and" National Advisory Council/Board members were 
similar— -over one-third were engaged in clinical activities and one-fourth were 
women (Tables ^, 5 4nd 6). . » " 

Delays occurring in recent years in making appointments to Natfonal^ 
Advisory Councils/Bo«ds were reflected in the'-finding that almost one-half 
'of the members of these ad^^isory grotips were serving the first year of their 
current appointment. In cont rast , , InSt itute IRG tftembers'^wefe ^re la tively evenly • 

> ' . • . \ 
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tpread among the four years-, while abput 'one-third of DRG Study Section ^embe 
were f irst-ywr appointees (Table 4) . • - 

Assessnent of tfii^Cttrrent NIH System 

All parts of the NIH review system did not receive the high level of 
approval elicited by the questions on the overall system and its general, 
fairness. Finequently, responses on specific features were less favorable 
than the answers given on more general aspects. Nonetheless, over one-^alf 
of the review group members assigned a rating of either excellent or gopd to 
the 48 questions oh the current system which are appropriate for this kind 
of an analysis, li That is, each aspect of the cBWent NIH review system 
covered by one of the 48 questions received a favorable rating from more 
than a majority. In f adt , 33 of the 48 questions on the current system 
were rated favorably by at least 80 percent of the review. group membersi 
The greater satisfaction of the DRG Study. Section members fs clearly evi- 
dent, as *is the relatively more critical assessment of the National Ad- 
visory (Council /Board members: 

Distribution of **fuestioas by the proportion of 
meniers responding excellent/good or the equivalent 



Percent of 
members responding 
favorably 



All 
groups 



Number of questions 



Initial Review Groups 
DRG Study 
Sections 



Institute 



(National 
Advisory 
Councils/ 
Boards 







Total *• 


48 


Under 


50 percent 


0 


50 


to 


59 percent 


1- 


60 


to 


69_ percent 


2 


70 


to 


79 percent 


12 


80 


Co 


89 percent 


14 


90 


percent and over 


19 



48 

0 
1 
3 
9 
11 
24 



48 

. 0 
0 
6 
11 
16 
15 



- 0 
1 
10 
8 
20 
9 



Persons accepting appointments and actively- participating In the NIH 
research grant review system may be expected to-be-f avorably disposed toward 
that system as well ^s knowledgeable about its policies and proceMuVes . In 



—Seven of che^'items in Part II of the quest ionnairli were oii|;Lt ted from 
this analysis — Question 4n; Questions lOb, d, f, h and k; and Question 11. 
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analyzing the ratings I2l assigned hy the review group members tb^ specific 
aspects <Jf the system, it^-is difficult to discern negative readEions.' With 
the exception of^thtee q^iestions, at lesjst 70 percent .of the respondents 
judged the current system as either excellent -or good.. HoweveR, it is . , 
possible to group the replies^ by the differing levels of support given 
specific parts of'^the current NIH research grant review system (Table t) . 

The strongest endorsements, those aspects of the current system approved 
\fy at least. 90 j^ercent of the review group membiers, included: ^ \ ' 

. Lack of general bias in the jllH grant peer revie^system 
well as in the performance .of ^fIH staff, peer review gro^up * 
members, and the peer review system procedures. The* 
^responses of DRG Study Section members were consistently 
the most favorable and those of National Advisory Council/ 
Board members consistently the least favorable (Table;^?). 

• • Lack of bias against minorities, young investigators or women 

in" the review of .applicatipns in recent years.' Again, the 
DRG Study Section members were the most favorable and the 
^ National Advisory Council/Bpard members the least favorable, 

htore bias was perceived on the other side of the coin- 
favoritism toward these groups, particularly ybung j 
^ scientists (Table 7). ' • * 

• • . The overall adequacy of the current NJH review in general and ^ 
for the initial review of traditional research project (koi) 
grant applications (T^ble 8). - 

, • • • The adequacy of the current review performed for the scientj^f ic - 
. and technical quality of new grants and for the capability of 
research investigators (Table 9). 

r y 

• »' The value of site,, visits to the review process and the quality 
^ of sit^ visits. In a shift from their usQal ranking. Institute 

•IRC and National Advisory Council/Board members .were *more ^ 
enthusiastic about th^vadue of site visits th^ the "DRS Study 
Section members wha con^h*ct such visi-ts much le\a frequently 
than Institute I RG members as part of their xevidWs (Table 10).^ 

• • The performance of review grou^ members as meaisured by the 
.quality of the discussion of' applications and thelj behai^ior 

during the review process. Once more, the^ DRG Study Section 
members were th^ most favorable (Table 13). 



^ — ^Rie questions on the current system permit the selection in most in- 
stances of one of the following: * excellent , good, fair, poor,* verys poor, or 
* no opinion. For some questions pn bias, thexhoic^s arjp: none, insignificant, 
moderate, «s igpif icant , very significant, and no opiniop.^ Percentages are 
computed on a base wjfifch excludes ''no opinion" and all noxyresponses to that 
question. ^' 
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. \ , . The scientific -and technical members' qualifications and per- 
' formance. The assessments made by the membcNrs of th^ Nation^tl 

• Advis^Q^jj^uncllp /Boards, which Include lay or public members, 
w « were less favorable than the opinions- of the members of both . 

\ ♦ initial review groups which ate entirely composed of such, 

scientific and technical members (Table 13)*. 

. . . NIH stalf qualificationa and performance in administering the 
' ^eer review system, both trhe ^^xecutive secretaries of the 
initial review groups and all other, staff (Table 13). 

At the other .extreme, the ^st support was given two of the current 
HIH restrictions governing applicant notif icatipn: The lowest rjtting (56 ' 
percent favorable) was given the present requirement which^ prohibits in- 
forming the applicant of the priority score assigned by the^ initial review.- 
group and the third lowesjt score (69^^percent favorable) was given the require- 
iaent which delays informiag the applicant of fhe overall initial review 
■group rertmmendations until completion bf the final review by the National ^ 
Advisory^ Cpuuc|l/Board XTable I). * ^ . 

.The weakest erfdorsements,' those aVproved by less than 80 percent dt all 
review group members, included: * ^ . 

.• . . Some bias toward- "cronyism" although rejected by over 70 percent 
I . of all review group' members who perceived no signs or only In- 
\' , significant signs t5f such favoritism ^.n the review of .applica- ^ 
tions* (Table 7). -'AbQut 20 percent reported observing mcrdetate 
amounts of sucVi a practice ^nd almost 10 percent reporSedj^ob- 
. serving significant or very ^significant amounts, Sharp differences 
existed In' the perceptions of "cron>;isin" among the review groups*. 
Only 5 percent of the DRG Study -Section members perceived 
significant or very significant evidence of "cronyism" compared^ 
with 1^ percent of tije Institute IRG'aetibers and 2A percent of 
the National Advisory CouncU/Board members! WithiiJ each^review . 
. ' group, women members ar^^^tlle younger '^scientists were the most^ 

- critical. . ' ' ' ' • ^ , , " \ ' 

. ^ I The reviw^ of^rogram /project And center grant applicant ions was. x . 
judged l^ss adequate by all threi types of review groups than 
^ . the highlyr rated initial teviewlof traditional research project 
grants. Howevpr^/ the degrae of satisfaction varied among the , ^ 
groups. Over 85 percent of the members ^6g^ the Institute IRGs, 

_ who perform these reviews and are most intimately familiar with 

the process, rated such reviews as either excellent' or good. 
Almost ^s hl^ a prop1>rtion of National Advisor:^ Council/Bpard 

, . \ meni)ers,.who ude the reviews in their deliberatlcUl^ Indicated^ . 
satisfaction. The^lowest rate of approvafl — 73. pgrcent — was by 
the DRG Study Section members, many of whom may not.be familiar 
with tills type of review as is evi^nced by their unusually ^hlgh 
35 ^percent no opinion/no response rate on this question (Table' 8). 



TABLE 1. ASSESSMENTS OF TflE (^mm NIH RESEARCH GRANT REVIEW SYSTEfi 

Crouped -by cht proportion of ail review group mcmliers responding 

exc«ll«nc/good or che equiv«lei{c i 



■f 



QUESTIONS ON THE CURRENT SYSTEM (PART 11 OF QUESTIONNAIRE) 

^ ! : f, : , 



PERCENT 



A. 90 percent or more rtspondlng excellent/good {19 tttms') ' 



^Ql,^ Hov *3equ«ce it che review currcncly ^rforaed «fot; ' * 
e.< The eciencif ic/cechnic«i quiAicy of new gr«nce 

c. Capability of rceearCh Inveitigmcors < ^ , 1 
How adequate in general, is che initial review of cradicional^ research project 



Q2. 



(^l) graac applications? 

Hov would you rate che: *' \ 

a« Value of sice visics co che review proceas ~ 

b. Qualicx of sice vis ice \ ^ & 

d. Qualicy of discussion of .applicacions durj^ng review sessions ' 

QS.T^Etdw satisfied are'y^u, on ^he avtrau* wich: ^ 

^f^. The QUALIFTCATIONS-^f scienp^ic/cechoical menbers of che review and 
^ advisory groups? v * 

'^c. The PtHIQRMANCE of sciencif ic/cachnlcal oeaber^ of che review and 
advl-^ory groups" 

Q6c,« How would y^ou describe che b^ihavior of ^och«r team oeabers 
review procSMs? 



i^uri 



ing Che 

caf^ who 



Q7, How .would you race the quallf Icaclons and~< performance of 
' adalniscer che peer review sysceia? * 

a. Executive secrecvrles of Iniclal rcv4.ev grbups 

^b.* All ocher icaM ^ 

Q9« Taking all signif icanc . f accors ln,C(MccounL, ho%( would you ^icharaccerl/e 
^ ^e NIH granc p.eer review syscen in cocb? 

Q9. On Jhe basis ^f YOUR OWN experience as a menber ■siitsii gftrop, 

howfirould you race che fairness and lack of bias ol- the NIH granC ^ 
^ peer review syscea? ^ - ^ 

a. In general " * > 

b. Wlclt resptfcc ^o performance of peer review group aeaberl^ 

c. . Wlch respecc co NIH fftaff perfon^anee 

d. . with respecc* co peer review syscem procedures ^ 

QIO. Please Indlcace- che e\ttT^t of sign£f icanc biases, If any, you observed 
' ,ln che review of applications In recenc x^ars ( percenc responding none/ 

». Ag« (anci-youch-) ' * ^ 

RacM^and-mlnorlcy) " 

e. Se»"(ancl-feBuile) 



96* 
92 

97* 

90 
'91 
91 



95* 

'95* 
^92 



94* 
91 

96* 



> 95* 
93* 
94* 

. 94*^' 

98* 
95* 



B. ,80 to 89 percent responding excellent/good (i4-items) 



< 



QK -How adequace Is tl)e review currently perfonsed for: 
I. i /ic^T^e-i and environment r^^ulred for research 

du.:aallic/ of che requesced period^f bi«6lg*c supporc ' 
04^ How would you race che: 

f. ^ Cf fecciveness of che syscea of approvaj./dlsapproval/deferral 

g. . £f fecciveness of che 1.0-5.0 priorlcy syscea 
1. Qualicy of suanury iicaceaeo^s (piolf sheecs) 

* At lease liaff of ^cbose Indlcac'lng in opinion selected "excellent" or 'Inpne* 



6i 

83 

88 
82 
86 



c 



TABLE 1. ASSESSMENTS OF-THE CURRENT NLH RESEARCH GRANT REVIEW SYSTEfi - Cont'd. 

: = r—^ ^ 



QUESTION? ON THE CURRENf'^YSTEM (PART 



J. Proc«|aMs i:tgsrdiQt* posslbl« -conflict 
RtvlSw of competing r«newiil i^llcacions 

ml 




questionnaire) 



Rcvitv of suppltacntAl ^plicstr^nsj^ 
Review of revised (kneaded) applications 



their pvogteik^ reports 



Q5. 



^How. set is fled sre you, oft che svereg^ 



with: 



fThe^eff^ctlvenesa^with which the NI« ^an| peer review system uses your 
^talents^.ind expertise ^ ■. ^ ^ * ^ 



. f.* NIH adainistretive polities procedures ^or ^eer review | 

Q6a. How' would you describe yout overall working conditions- duvlhg the revi^ process? 

QIO. PleMe Indicate the cxtetit of si^lflcani! J>lasBS» If any, yov have observed In 

the- review applications in recenP years ( percent respond4>n^ iv)ne/ins^njfic.ant^ . 

4r. Innovative ideas (anci) ' ' *■ , * 1^ * * 

i., NIH staff l;jtetfer*n&e ' . # ' ^ * Ai 



PERCENT 



89 

87 



a8 

' 84 
87 



89* 



C. 7(3 tp 79 percent responding excellent/^gootr (12 itemsl 
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Ql . How adeguaee la the review cyrre;ptly peffftmed for: 
^ b. Collaboi^cive ar-rangem^tts es9i»nK tal. to research 
e. ^ AppropVlateness of re^e^ted «bud^t , - 

How adequate, in general j|^s* the Initial review of (program 'project '&nd center 
grant applications? > 9f • ' ' 

Ho^ would you rate, the . > . .'^t 

c. Time ^vail lib le for si^e visits 

Tlae available •for review 

How saii^ ^ 



0 ^ 

h.V EHff 



Q5 



J^ed 



veness of .che ^00^00 priority score ranking 4yst^iii ' * 

Ied are you, orv the, aveiage, w^'th: , 
d. pie q^aiiricat {ona and performance of ^pubj.i^membe'r!^ o£ advisorry groups 



"^W^dequecy o? the 'select ion process for peer review group members 

rki 

erms of th^f^qu'acy of the facilities provided 



•O^-^piltt' vo^^iyou describe^ )^r 'working oton^c^ons during the review process? 
* b. In fer 



^Q^Oi. 



Oi2c. 



'^nabU.fng you t^ mincpion at effective physlc^/mental levels (ImpsctAf 
. fatigueV ^ ' ' V ^ • ^ 

indlcfCe che ex^nt d't cr^qylsm, If ai^, you have observed in the 

r«view^3f applicstiohs in.recent yrfrs -^ Perlbnc responding ndne/ 

Ins fjtfflf leant ) ' ' v $. , - - ' 

Indicate your opinion of the pre'sent NIH requirement , that the research 

o#l 



grant applicant NOT b^ Infonted o^lndividual reviewers-* t^oomen^s^ 



?9 ' 

27 
71 

72 



50 



tp/69 ^rcent respoy^^lng excc^lTentXgpod ^3 M^s) > 



Qlg. (low adequate 4.9 thc;xevl«.w ^currently per,fory|^d. for r'elevance of ^'proposed A 

"^Z research NIH programs * ® n - 0 ^ ' % ' 

' " ■ ' . * *^ 

Q12. Indicatti. v^ur opinion of/ the prj^ent NIH. ij^^uirement that research grant 

apVAcents NpT be inforlaed of ' 

a. Ovcr«llTnitlal. Rftvi-ew (itioupi^recQBBDeriif^f tlons u;it^l ^fter fln|^ Cdtincll/ 
^ Sv>ar<l revi^ ' , ^ ^ ' ^ 



b.' Inlct«l Revt*v Group prlqrlty scojr/s 



67 



69 



* .At least r.alf of chos^ Indicating an opinion ^elected "encellent" or 'hione". 

- * . ^ Nd'op'lnlop/no reapo<jaa^rste w^s- l? to 2t portent. • » 

2 Ho oplnf<j^/"tjo restTortse rate^waa ^2 percent. • . ' ' ^ 

63 - ■ ■ ' 
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The adequacy gf the review for program relevance » which Is phe 
• primary respocislbility of the National Advisory Councils/Boards, 
was assessed as excellent xj.t -good by '67 perCent pf the review 
group members » the second lowest rating for a specific feature 
of the current >i:evlew system. - DRG Study Sectlontjnambers, who 
arc Instructed hot to .concern theuaselves ^wiCh thi^faspect of 
ihe review, w^re the least ^favorable while Irtsc^-tUte IRG mem- 
/bers, some of whom may consider this as*^ part of their own func- 
tion,, were the most favorable (Table 9). Within all three types 
of refvlew 'groups, menfcers currently engaged In clinical activi- 
ties were leSs satisfied tt^an those not so» engaged. ' ' 

The, reviews for the appropriateness of the requesteji budget and 
<fir collaborative arranj^emerats essential to research Vete per- . 
^ celyed <io3?e favorably by members?" of both types of initial re^ * 



view groups, who are responsible^ f or Mihese funptions,. tjian by 
the'meiffl).ers of the National Advisory Councils/Bdards (Table 9). 

r • ' 

, The^lme avallablfe for slt^ visits was rited least favorably 
2by the membSrfe 'of the Institute IRGs, the review group 
that;^ is most concerned witji such visits (Table 10) • How- 
ever, only 5 p^?cent*of this group rated the time available, 
-as either Tpoor or very poor. ^ ^» , • 

Th€ lD0.£o>500 prlarlty score. ranking system apparently vposed^ 
^ a problem of utlderstandlng to members of both types df Initial . 
^tevlew groups a,s a high percentage' did nqt respond "to the ques- 
icli^ on dhe effectiveness^ of V this system. However, about 
- three-fourths of all thps£ responding .f rom each of the* thiree 
review grdup^' considered 'the system as either excellent *or 
good, (Table 10). . ' ' 

Current restrictions on applic>ant notification had s^lzeable 
opposition. The^ present NIH requirement fotUidcllng research 
grant applicants to be*informed of, their prioritfy scores was 
pexceived'as a poor or very poor practice by almost one-third ^ 
of all review group, members. ^ Approximately one-fifth of. the 
members also o^jecte^. to the other two j:urreiit requi^rements 
listed — not informing tkeapplicant of the overall IRG reconi- 
mencfation until afte^ flnaT Council/Board re^view and not making 
'individual reviewers ^cpmmehts available' to app>f?ants. However 
a majority of all ^view gfoup memjtfiers^ regarded all thrise 
requirement^ as either excellent or ^ood (Table liy. ^ >. 

The^^me available f^r reviid^ appeared tQ^be an area of* some 
dissatisfaction. About 70 gerfcetft of all review group mei^^Vs . • 
rated th^ time available as- e4^^|^^9^^^^"^ good. However, 
only ^pel^en't fe^t sufficiently llMtatisf led to rate the time 
avaiJ.aD lie V^r "review as either poor arr' very poof^ *The greatest 
dissati^aeCion V^s shown by t^hememer^^f the ^fational Ad- 
visory Councils /fttfarclrf (fatle^n) . , , ^ / \ ' 
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Workring conditions as measured hy the- facilities provided and 
the provisipns made tb| exiatle the members to function at ,ef- 
fective physical and mental levels were felt/ tb be much less ' 
satisfactory by DRCf Study ^Section members tjfen py the members 
of the other ,two revieirl groups. Members -of tine National Ad- ^ 
visory Councils /Boards were' very well satisfied (Table. 12). . , 

The selection process f^r ^peer r^svlew group members wks not ^ 
heartily endorsed — less than thrae-f otfrths of all review mem- ^ 
bers rated the adequacy of this process as excellent or good.^ 
'In addition,' the substantial no opinidn/no. response rates among 
all three types of review groups implies that many members may ^ 
feel uninformed (Table t3). * , ' 

The public members' adequacy and performance was rated as ex- ^ 
* cellent or good b^^ slightly ;oveif tVo-thirds of the National * " - 
Advisory Council/Board members. ^ very high p^portion of ^ - * . 

initial' revi'ew group' members supjilied no* opinix>n as tH|^ prpbabiy ' 
had no basi$ for fqtming any* opinions concerning the' qualif i^a- ^ *^ 
titfns or. performance of the public members who are 'not represented } 
on their groups (Table 13).. Ai^ong the National Advisory 
Couifcil/Board .members, opinions appeared sharply divided in , • j{ 
such a way as to lead to the conclusion that the public -memr 
bers were rated less, favorably by th^^scientlf ic members than " 
by themselves;^ ^ ^ ^ ^ 



ChAiiges in the Current System - , - - * 

The review group membert^were generally negaj^ive in their reactions to 
Suggested /Changes for "opening up" the present NIH research grant sys- 
" tem» Only four "of the 16 recent, or pbtentiai changes listed in the queslfion- 
Xiaiih werer Derceived by^ majority of member's as .probably having a ''very 
favorable" ^ "favorable^' impact on the quality^ahd efficiency- of the; 
syjstem (Table 2). " In^ addition, one recently adopted change — opening* re- - 
* view group seSfSions to the ,publib when not reviewin^apjplications or budgets — . 
, was considered by a majoSrity as havlag either a favorable impact or "no - 

. ^ Of the four suggest4.ons for "op'etiing up" the system whtch the 

oajori^^Of the survey resgondfents considered desirable, thtee invoiVe^ ^. 
Inoreaslnj the access to the siiipmary statements (pink sheets). Making/ "'r 
tt^^ese' sfat^ment^ available upon. request to. review' group Wmtyer3 and. Without 
•the priority score, to prtpcipal investigators drew considerable support; 

^ less "Enthusiasm was shgwrv t^}ward making the sturanary statejnents available 
po principal investigators with th^ priority si^ore. »The fourth suggestion 

*J^ceiving majority approval .dealt with making |J^«lj|;iqu68 of the individual 
^^ftvieVefs available to the principal invest iga§^ i^^n request (with the 

^ 'reviewers anonymous)/ ' • 
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TABLE 2. ASSESSMENTS OF THE IMPACT OF RECEWTJ^ND POTENTIAlr-CHANGES 

\ ' ■ 

__firoup€d by the proportion of all review group memljers responding 
very favorable/ favorable 



Questions on the impact of recent and j)otential changes 
• (Part III of questionnaire) 



Percfent responding 



Very / 
favorable/ 
favorable 



No 
effect 



■ ^ : ' '■ ^ ; ' ^ 

A. More than 50 percent responding very favorable/favorable (4 items) 


- 


Q6. 


Makittg summary statements (pink sheets) available upon request 
to: . ^ ^ ' ' 
a. Review group members 
^b. The principal investigator*, without the priority score 
c. The principal investigator, with the priority score 








83 ' 

79 

60 

w 


12 
8 

7 


Q7a 


Making the individual .reviewers * critiques available to the prin- 
cipal investigator upon request, with the revieweib anonymous 


53 


6 




B. 20 .to 49 percent responding very -favbrable/favorable O'ltenfe) 




^1 


Release of new, funde'd grant applications to the public 


2k 


25 




* Opening the sessions of the Initial Review Groups and Advisory 
Councils /Boards to the p'ublic when not reviewing applications 
or' budgets 


34 


37 


q6d 


Making susrnry statements (pink sheets) available upon re^ulst 
to ofiic^ials of the applicant institutlooi 


- 37 


10 



C» Under 20 percent responding *yery favorable/favorable .{9 it^ms) 



13 


18 


7 


\ 4 


9 


: 8 


5 


5 


8 


• ? 






11* 


9 


' 7 


K 


^2 ^ 


13 



*Q2 

Q5 



^Public release of all renewal/contin\>ation applications (funded 
or not) . 



Fot" those sessions whiAol) review grant applications 

a. Opening Initial EUrvlew Group session^ to applicant 
investigators ^ ' ^ • ' 

b. Opening IRG sessions to oSkpfals of applicant institutions 
c\ Opening Initial Review GW^! sessions to^ the public 

d. i)pening ^Advisory Coilncil /Board Sessions to applicant/ \r i 
* i Jnytltig^torlj ' ^ ^ ' ^ * ^ 

e. lfcpening Advisory Council/Bpard*^ sessions to officials of 

applicant , institutions 

f. Openfing Advisory Council/Board sessions to Che publfc 

Q6€ Making .suiomary statements^ (pink sheets) available, upon request^ 



Q7b 

'I 



to the public ■ * , 

M^ing Bhe individual reviewers' criti5ues available to the 
.Jpriocipal investigator |pon request, Nith the reviewers 
identified " * , . ' ' , 



* Prartices already in effect at the tiAe of the survey. 

^ ^' ^ < 66 . • ' 



Even the three recei^iy instituted changes toward .-"dpenihg up" the * 
^system were expected to afrfect'^iAe NIH -peer revi'ew system' advetsely. Ap- 
proximately 70 percent of all review group 'members anticipated negative 
.effects fT<m the public release df renewa^ or. continuation applications; ^ 
one-half expected unfavorable effects frdto the public release or new, 
funded gratit , applications; and 30 percent- regarded Opening the sessions to 
the puBlic when.not reviewing 'either applications or Budgets a^having a ^ * 
very unfavorable or 'xtnfavorable .impa1c*6. - ^ ^ ' ' ' ' 

Over 60 percerft the review group membrfrfe <were not aware'of ajl^ ' 
three of the recently dBopte^ changies toward "opening up" the NIH review 
^^System an^E^ substantial 17 percent were* not aware of ^y of thgm (Ta&le 14) * 
^Such a lack of awareness. on the part of the review group members raises the 
qitestion of adequate communication. K mitigating factor may be th*^. fact 
that many memberi^ are' serving in the first year of their current appointment. 

Another ^¥rt of the* questionnaire lUsted 2Q suggested 'improvements ^and' 
requested the ^vlew group members to rate ^hese Siiggestions.^ The tatings 
designated by the' review group members indicated a general satisfaction with 
the current sy«em. Z2 Of the 20. suggestions, only ^one— ^dealing with reducing 
delays—received strong support. Over -three-rfourths of all members en-'> 
dorsed a suggestion that a special study be made on how to shorten. the time 
between the submission of an application and Its subsec^entiJ|f unding or" denial 
of funding. None of the other suggestions received approve^ ftom a majority 
of review group members .(TabJ^ 3). , / 

, Seven of the 20_suggested i^ro>>emetlts were considered as either^ ex- 
cellent or vfery good by from 29 to* 4*3 percent of the review group members.^ 
Of these seven suggestions, .tfigree dealt with reducing, the workload of ^ initial 
review group timbers.. The method given the mo$t support was to increase -tbe 
numl>er of review groups; over two-thirds rated this suggestion as fair or 
better. However, 'the other ^o methods si^geste|l — r|yucing the. maximum 
term of, appointment Xio ^ree yeai;s, and staff screening and rejection of 
inadequate applicatidns^were rated poor or very poor by mtee than a malorii 



poor 

. ^. \ ^ 

- Differences among t)ie^ £h,ree t)ipes of r^vfe 
variations in^th^ir support of sdme of thesug 



ups became a 
ed im|frov^metits 

all thr^e groups considered the special study oh 'reducing delays. 
8irable»y ; a majority of Study Section ^members did not rate to; 
other 19 suggastions as excelle^nt or very good. However , /a-majo 
Institute XRG outers thought encouraging reflew group members 
site visits was/at\>xcellent or very good idea^^^as did a majo^tV of 




ijority. 

in.th^ 
though 



/3. 



ommend 
National 



— An analy^s of . the Comments ,section of thfi qutstionnalfce may indicate 
substantial support for improvements which %^re not listed in] the question-* 
naire. Such an analysis will appear -in -Pa^ II of the R^port^t^o the Survey 
Lttee. • W . • A . > , , 



Sul}C(>mml^t 



TABLE 3. ASSESSMENTS OP%UGGESTEDi IMPROVEM^JITS. 
'^^Ranked by the propoption of all review group members responding exceHent/very good ^ 



Percent responding 



•Qi^estions on suggested inqirovements (Part IV of quest ionnair^ , 



Ex(;ellent/ 
very good 



T 



Fair 



l7*rA special study on how co shorten the time b&dween -submission 
of an application^ and funding or denial of ^fundirtg ' ' ♦ 

Q2^. Reduce the workload of IRG ^mbefs by increasing tfie 'nui^r 
of review groups \ 

Encourage _review*group roembe.rs to recommend site Visits 
Provide specific structured format for individual \ 
reviewers* reports ' . m 

|-Q13. Permit applicant '^o' sug^st revieweiss in Jase of "^peal only 

Q8. Increase the number of site visits' , . . • ' 

Q2c. ^Re^ucb the vor<it<ad of IKG meraiers by redating the maximum 
tern of app^^tmept to 3 yeai;s » 



Q9. 

■ *Q6. 



p^it 

work 



Q2d. Reduce the Jorkload of IRG members by staff 'screening* and 
^ rejection of inadequate apjJiicaticfns ^ * 
^5;\ Provide specific re^ie*rf format with' Weighted r»C ing^ procedure for 
♦ ' review criteria- - ^jjij^ ^ " " ^ 

Ql 1. Delay,, the appoistmejit of a retiring member to another group for* 
^ . I longer thftn the one-year moratorium' now required between ^ 
appointments ' . 

Q3. Increase^ the assigned feviewe^s for each proposal from to. 

. 3 in each IRG * . * * 

Q2^; Reduce the workload of IRG members. by, increasing the number , 
I of mei^ers' in each group ^ ^ - 

Curtail or omit .4iscusfeion pi those applications* judged 
*by assigned reviewers to be eith^" outstanding or very poor; 

Q12. Permit an applicant to suggest* reviewers . ^"^^ 

laclude sclent it ic/techfii^ca I -members employed by for-profit, 
organizations on'grant review flH^ 

Decrease the number of site visits 

% 



OA. 



Q14. 



^Q15. More frequent review group meetings 
(JIO. Disco^tinije site' visits on research project* (ROl) applications 



QI$. l^s<i f req-uent -review group meetings, 

^ t - 

Ql, Increase the ^orkjpad of IRG merafeefs 
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33 


'24' 


31 
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25 ^ 


20 


23 


21 


23 
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19 
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12 - 
5 ' 
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/I No ^inlcn//o response rate was 15 to 27 percent. 
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Advisory fcounail/Board.membersT^ In -addit ion , more than Ojjierh-alf of the ... 
Nat ionalJAdvlsor.y Ctmncil/ Board members 'thought i't would be an ^'xcellent. or * 
very. goj*d .ide^ to ha\\e a jspecific stYuQtiared format .^r ^individual .reviewer's 
reports, to have a specific review fortnat^vith weightecl rating procedures ,fbr 
.the ».r^ view criteria, an,d to reduce the workload of 'initial revicjrrgroup mem*^ 
bers by increasing the number of review groups* (Table ^6). ^ . ^» 

More detailed discussions of . specif ic'aspects of - the current nIh -review 
system and of suggested changes or improvements ^re^gfven in the . last three 
sections of this' report. In addition, certain significant aVpects of the* 
cmrrenl?^IH review system which have-not been mentioned Jj? this -section be-/ 
cause tKey'd'id noc fall into edther\the '"inosat favored" or ''le'a^f /avored'* 
groups ar^^aealt w^h jn some .detail. ;These ihclude all those! aspects pf 
the^current system that were rat^d as excellent /good (or the e^ivalent) 
by fpDm 80 to 89^ percent of all review group members (Table 1); < 




PEER/RgVI 
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SELEcnorr or. initial review caajp-MEMBERS 



Ut is clear, from the pQblic hearings, letters, and discuss iois with other 
interested persons that: (1) the selection of initial review group (IRG) 
meobers is a^saisitive issue of cons uiSr able, interest; (2) there is a strong 
desire for more formal input from 4he#fscientif ic coitinunity arA/ot public ^ ^ 

* regarding noroinaticxis for mentor ship^to IH3s; and (3) the method by which * 
iiiaift)er& ^are selected and the many cmsiderations involved therein have neither 

*been "^adequately described nor are gaierally understood,, ^ ; '/ 

I. BACmOUND . ' ' . 

Bie areas of biomedical science for '^licii.a^ particular initial group has , 
review responsibility are briefly described in the HEW publication "NIH 
Pi^ic Advisory Groijps, Authority, Structure,* Function, Meibers". (1) A 

* detailed and specific description of .the revi^ responsibility of each IRG 
in th6 Divisicxi of Research Grants (ORG) is further provided in the 
Referral Guide related to Study SectiOTS (IRGs). (2) The latter is updated. ' 

. annually to refledt advances in the field and/or «tainiArative changes in 
review rei ^yn^ ibility of an IRG. In additipn to providing, technical merit 
revie w o^ ^pl'l^^t ions , menbers also have the responsibility of surveying, • — 
p^-flrteHfti^c leacters, the status and needs of research in theii* respective 
fields • ' ^ 

II. REVIEW OF CURRENT PROCEDURES, RESOURCES, AMD SELECTION CONSIDERATIONS 

A. Current Procedures 

^ ' Formal nominations for menbership to an initial^review group, originate 

witlrthe Executive Secretary. One candidate ig nominated for each " 
posimon an* the final selection is made by the Director, NIH,^ or his 
designee. The factors considered and resources available 
to an' Executive Secfet^try in making these selections are discussed in 
'subsequent sections. * / . ^" 

A. series of wall defiped procedures are f611owed,' and review of all 
^ - nominations is conducted at several levels* w itj^ n NIH and H5W before ^a 

nomination is approved ^arid ah. invitation to SBwnentiership^is extertded by - 
the Director or ^ehcy^head•* ' — 

1. Ihe Executive Secretary initiates a nomination by forwarding the 
, , "Request for Approval of Nominees for Public Advisbry Committees" 

(Form HEW-532, Attachmoit 1) to the Corrmittee ManagCTi«it Office(CMb) 

the particular Bureau, Institute, or Division (BID). 'Jhe 
information provide4 includes a description of the candidate's, 
special ^pertise, type of qualif icajtions needed 'for the position, 
prc^xDsed'term of marbership, and the nominee's curriculian vitae.' 

^"^e Director „ National Cancer Institute, has ^peqial separate statutory 
authority to .establish advisory connittees, 42 U.S.C. 282. 



sic ^' 

^ -J 



2. At the sasoe^im the Executive Secretary also forwards the 
list -of nqninees aj>d the above informatfion to the Chief of Extra- 
mural Programs (or other designated person) of the Institqtes for • 
which the IH3 reviews substantial numbers of applications. Ihe 
covering memorandum notifies, the relevant* Institutes of the 
.proposed appointment and provldes^-i^Jportuhity for informing the- 
Executive Secretary* of any serious questicMis or concerns regarding 
the nominations. 

* ' 

3. After t)ie naninati<xi(s) has been reviewed by the BID Directq^ 
or 'his/her designee, th^ Coritnittee Management Office (CMO) <^ 
initiates an Availability Request Clearance ^vrfiich is forwarded, 
through the OO of the Office of the Director, NIIO to the- relevant 
office at HEW. There, informaticMv concerning previous ffietrberghip : 

HEW committees and terms of office is provided and considered. 
Ihe total time required for this action j.s six to $ight weeks. ^ 

4. Prior to the issuance of an invitation, the Executive Secretary 
has a final c^jportunity to decide whether the specific nominee 
should be asked to serve on the IRG,^since during the period 
between nomination- and approval,, ment^ership^requironents and/or 

-ccxiditions relating to the nominee's availability for service 
may have changed. 

5. The BID Director or his/her designee issues the invitation. to ' ^ 
serve, on the specific IHG for a specif ic terra^- in a letter vrtiid) 

. states the ifevie^w and other functions of the IRG and describes 
the responsibilities indumbent upon such manbership; Acconpanying > 
the letter of invitation .are the required forms, including the 
Confidential Statement of Employment and Financial Interests (HEW-474 
Appendix E-2) for determining potential cOTflict of interest. ' 

6. ^ When the letter of acceptance and the conplet^ HEW-474 form 
are repeived, th^ latter i^ reviewed prior to final appoihtment. 

^7. Nominations for* service as ch^irpeiteon of an IHG require, in 
^ addition to the data supplied^ for regular meanbership,' a fllemor^indum 
describing the noroihee's special qualiticaticjis for this position. 
Sitic6'pr;ior .service with the l^is desird^||\jfcis selection is 

frequently made by the Executive Secreta^^nMh^^mong tfcW current 

. members. In othe^cases*, the nomin£i|A^^^ re^ippointment' 
of a former mesrter\>f the 133 if trf S J^^^^ggi^HM and justi--. 
fidation is required for this, aStitH." 

B. Resources 
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The^naiAnati<^ and selection of menb^rs il^a majpr prof^i^ional 
responsibility of the'SlExecutive S^cytayy,' involving long-range pi 
suggestions, anfJ information from many different .sources.* • It is. a 
of cwtfnuous research and concern. Numierous resources are availabl,e 
are constantly drawn upcm by the Executive Secretary in developing f 
suitable nominatidiis for membership: 




1. Hie Executive- Secretary's own knc3»?leQge or jthe discipline and 

of the scientists who are making, significant research contributibns 
to the^il^d, _ / • , ^ ^ . ^ ' 

a. Review of NIH af^licant ^nd.grantfee files and sunmary 
statenents ^§ weli! as curricula vita and publications of- , ^ 
invest igateoB. ^ \ ' • ' 

b. Reading of the major scientific journal^awl publications 

in the field. ' / ' • 

c. Attending relevant professional meetings. Hiese provid^^ 
— ^r^n valuable additional way of keeping informed of the mj^t 

significant new studies^ .in the field and of identifying the 
♦ investigators v*io ar^ doing th^ research. Knowledge is thus 

gained of the interests and exgertise of potential future 
IPG mejrbers. Such meetings also provide oppprtunity for 
consultation with eniircnt investigators v*io are not, or 
, _ ' have not been, maitoers of the IRG,group^ 

2. Service of investigators as "special reviewers"; ^dr example,; 

' (a) as ad hoc c<msultants ^ a regular IPG ranting wheruadditional 
' or special expertise is needed? (b) as members of a prdjfect site' 
visit team; or (c) by providing written collateral opinions on request. 

3. Solicitation of names of outstanding investigators in a given 
area from former and.ciitrent ^RG members, xind* other leaders- in the 
field. ^ . > * ' ^ 

4: Lists of women and minority scientists. ' ' , 

5. Consultation with scientific and ^professional sSfcf^f the' ^ ^ - 
vatious BIDS who havfe wide ccMitacts with.^the scientists itra given ^ 
field. I ^ ' ^ * ' ^ 

6, intending, as observer, the review of applications by other 
agencies— especially t^iose having review panels in related disciplines. 

' ' - ' ' ' ' ' * 

Frcm a cottoination of such sources the Ex^utiye Secretary develops a, 
large backlog of naines of well^^lified scientists as potential JPG moitoers. 
Ncnuinees are selected .from this list. 

C. Nominati^ and Selection Considerations 

The criteria, guidelines, and conditions for i^Iection and 
appointment of mait)ers of Nl4l Initial Grant Review JGroups have undergone 
* chafes and evolution since the peer review system was e3tablished. ' ^ 
^These are outlined in the memorandum to BID Directors dated 6/23/72 
C (Appendix D). More recently, proposed rules governing the cotposition 
of peer review groups (as published in the Federal Register , Vol. 41, #61, 
p. 12987) (Appendix F-2) are in the process of being. codified. ^ 



. / ' ; 

In preparing a slate of norninees (usually three ^o five, one for 
each vacancy)#as replacements \and/ojf additions to the nieinbership, 
.'COiibination of specific q|iteria an&^ conditions f or appointmoit ^st be 
considered by the Executive Secretary relative to the present conposition 
of the IHG , Sane of the , major cons i-der at ions involved are 1 ist^ below - 
to en^ailze their importance and interrelationships. 

Areas and disciplines to bQ covered . Since thefprimary function 
of any IHGls to provide expert scientific inner i-t rev-iew or the 
highest quality, the first and^ major consideration ijn selection of 
roenbers is the 'particular area(s) of expertise needed, ^is relSi 
not only to imnediate or current needs but also to areas in \^ich 
deficiencies will occur as other menbers cocrplete' their terms, of 
service. In general, ^ investigators with broad knowledge of ttifi field,* 
as well is excellent expertise in a specific subspecialty, are 
sought. Hie inportance and the complex itjr 6f ^ is- fu^t ion are ^ 
formidable. Consideration of appropiJiate nonin^s fori^ach opening ^ 
requires not only the selection o^ people in specific disciplines 
but a search ^or and recognition of expertise' in SjieciEic ,subdisci- 
pLinary ar-eas * - ^-balanc^ of e^jpertise in-such -at eas in the^S'entire - - - 
committee is alsarec^ired as is an understanding of the way in 
which the appointmwt of a specific individual would affect the 
specttum of expertise — not o^ly initially^-HDUt v*ien other monbera 
retire. Specifically, this exercise requires this* assessment : 
over future years to assure the necessary balance ^d coverage 'for ^ 
each new appointment proposed to the Initial Review Group. This* 
function makes the greatest denands on the professionalise of the 
Executive Secretary and is perhaps the single'mo^ inportant ; 
^"dfeterminant as to v^eth^r the IRG will operate efficiently with 
the appropriate balance and expertise. 

Previous service, if any, on scientific y^rit review groups of 
NIH and/or other agencies . Before a nanIi\^tion is ^en^ forward it 
must be determine^ that the nominee is notXcuff^ntly a member of an 
1I5W coninittee ana has noli served on such for at least one year "prior 
to the expected date of appointment* It is the policy not to 

•make reappointmerfts to the same IRG group* Exceptions to these 
policies are rare and are made only in special cases. For examples . 
(a) to appoint a chairperson of an IRG, since previous service on ^ 
that review commfCtee is valuable experience in undertal^ing this 
special responsibility; and ,(b) VAien the rtumber of experts in a giv^n 
• flel4, or subspecialty is so limited as to require reappointment* 

-Other rare exceptions may involve dual appointments, v*ien the^ ^ 
nominee is, for example^ serving on inpther NIH Advisory- Committee 
such as the Board of Scientific Counselors of -an Institute which 
reviews only intramural rei^earch activities. ^ 

, Institutional Representation^ on' Initial Review ^Groups . Individuals • 
nominated for membership musfcie drawn from institutions not cirrently 
represented on the IRG. Two inSlviduals from the same- institution 
may not be appointed to a given IRG. 



4l Geograp hic Distribution of Monbership . An attatpt is rhade to^ 
Istve representation from all areas oj the U. S. and to. maintain a 
balance Vlth regard tc^ geographic dis1;ribution in the nomination of- 
■ new manbcrs. ' Again, this must take, into account the location of 
current meitioers and the time wtten each individual's appointmen;^ends. 

5.~ Minor ity Representation . Through the various t^sources 

mentioned above, there is -a continual search both for ethnic 
group and wcnien scientists who a>e experts in specific subsprcialties 
'that rfeed to be represented. Equitable representation of sudi 

■ ■ y- ■ - - • — — - -- J — — i- — *- -'-"'jideration 




Term (&f. Appointment .' Although the' appointments to an, H«3 are 
• usually £or V four-year term, manbers- are sometimes nominated to « 
serve for only two to three years for various reasons. In this 
regard, consideration is given to such. matters as t*Te nominee s - 
other professioiWcoraititments,. future plans for sabbatical leave> etc. 
A further factor' is the number of member's scheduled t© conplete - , 
their terms of service in a given year, to maintain a desirable 
balance between .experiended and hew'menibers . 

7. Persona l Qualifications , less tangible,, but nevertheless 

essential additional considerations relate to judgment concerning: 
■(a) the nominee's ability for functioning effectively ,an4 cooperatively 
^ 'with the group and wM the Executive ^Segretary; and fb) the nominee s 
"'^ conscientioufenes? in fulfilling the ohligatioiis incQmbent upon IRG 
T*" membership, and for carrying oi^t the Sbjectives of tfte Peer Review 

System. ' ♦ ' . ' m.^ "* 

ill. SU!3GESTtONS FOR CHANGES tN THE'gELECTyN OF IRG MB<BERS „ 

The Study Team's interactic»is with /the scientific comtunity and with 
the beneral public left little doubt that many individuals are concerned 
- -about how -IRG members arer-or appear to.be— selected. Many of those who 
coraneoted on the present grants peer review system expressed the t^liet • 
that the pool of scientists from whic^i NIH draws reviewer^ :is unnecessarily 
.* and inappropriately narrow and thSt many qualified individu^s consistently 
^ • are over.looked. Others expressed uncertainty and uneasiness about the 
seemingly closed nature of th^ reviewer selection process and urged both 
■ iftor^ openness and broader p^ticipati<^b/ the scientific and lay , 
comnunities. It is' also apparent from the public hearings, the cowente, 
and 'the letters received that most individuals do.rtot know how-II?; ,members 
* are a^lect^, nor are they aware of the care and wide variety of resources 
currently enployed in making these selections. Thus^the process described 
' - above seems to be one aspect of the NIH grants pfeer review aystem^ 

in need' of clarif ic?rt:ion and-could benefit from input from kndwledgeablfe 
^interested groups •and/or individuals who are, not now ^eg.ularly) consulted. 



Given the widesp.Fi/ad concerh aimt go fundamertal /an eVement'of the 
.grants Fjeer reM^iew systan, the Study Team concluded tfist the NIFT should 'I 
make the details of its reviewet^election procedures ^ore broadly 
knOKm ,and^ In additioh, should provide* a mechanism whereby individuals 
and organization^ qan suggest candidates for IRGl membership. 

current practice of obtaining suggestions, for "repl^icements" V 
or for future membership, from retiririg or current members, has been 
challenged frdm various sources ^ on \;thB basis that IRG m^nnbers tend to 
perpetuate membeirship''f rom their own institutions or from among their 

' cblleagues. The perception is that meittoers choose their successors, or 
that, basically^^they control who is invited to serve on IPGs. This 

jlnisin?>ression can and should be correctjpd without delay, byt (1) mc^, 
wider dissenination of information regarding the range, of resources 
currently dr^^upoh by Executive Secre^ries of IRGs;'and (2) .publication 
of the criteria and conditions, for' seleC^t ion (as outlined in Appendix D). 

Another suggestion .has been made that current and retii^ing members • - 
should be specifically excluded ' frdm providing aQyice regarding replace- 
ments while they are still , 'officially, ~ ntpnb^rs of an IRS. Adoption of 
jsudi a policy would, hc^fully, aid in correcting the perception that a 
••buddy systQTi"* of selection exists. Rowe4ei|, while this may be desirable 
for "cosmetic** xeasons, it would weaken tWe con^irjued effort to^jjnprove 
the quality of review'^ to Obtain advicfe f rW highly knowledgeable ' 
.sources. Those who are serving on a par^'cular' IRG are often in the / 
bfest position of all potential consultants to Know what the gaps in 
technical expertise are or ate likely, td be, and tOi)rovide suggesticms 
as to the t)est qualified scientists to 'fill these rteeds. It would, 
therefore, (je in the .best interest of ttte maintenance of thfe quality 
of the Peer 'Review System to continue to seek suggesticxis from current, 
retiring, and "fotmer IRG members' as' well as from a wide^ variety of otiier 
sources. . . ^ 1 

It has been suggested that ah IRS Executiive Secretary provide- a . 
firsthand second nomination for each position. to be filled. Presently,- 
ronly one individual is nominated for each appoirrtinent ' to be made. 

The possi&e advantages of havihg two noilihees are: (a) if the first- 
dioice nominee, is unable to accept; appointm^t there would be less delay 
in "filling the positicm; Jb) there could be more involvement of other 
interested parties sudi as staff of thsj various Bureaus, Irjstitutes, and 
Divisions (BIDS) of NIH; and (c) the seijonld nominee may help dispel the 
^n^ession held by many that , IRG and Council members personally select 
their own replacements and that IRGs and Councils are closed, seleqt 
groups of selfrperpetuating cronies representing and distributing 
research dbllaci^only to .the largest and most prestigious institutions. ' 

Disadvantages ax^ (a)' since the combination of conditions which must* 
be met ^presently in the selection of only one suitable, available, and wel 
qualified ^petf per vacancy is a difficult and time-consuming undertakii 
this would gonpound the' problem;' (b) since the scienttfic expertise, 'th^ 



lodatioh or university affiliation etc., of any one nominee is 
•siqnificantly related to other nominees in sterns of^scientitic 
requirements IWI* other conditions to be met, , it cannot be 

• considered, in isolation from all other hominations or acceptances. 

It is felt that the disadvantages outweigh the advantages of Providing 

• two-nominations, simultaneously for each position since this would add 
SSni^cS^tly to the background work of each Executive Secretary while 

' qi?!™ I^Ue added flexibilitjf. If the pooi.of, known ^f^^^^f^};'-'- 

• ll^s can be substantially increased by other mithods, it appears 6^at 

• Sle rSt process of selecti^on of one nominee for each vacancy sh&uld 
be continued. 

• Finally, the su^stion has been made that -investig^ators «»ay subnit_ 
• Jf'bossiblTconsultants who, they feel,' are Particularly knowledge- 

^ if thftechnical aspects of the research proposed. This procedure 
■ is presently in effect but obviously it is not generally kno*m. It is 
clear th^t a H^thod must be devised to inpiement this procedure m-a 
formal manner.* ^ _ * ^ 

JgiG0W1ErB)ATIONS : • ^ ^ • 

1 tHAT IN ORDER TO MAKE M REVIEWER SELECTION PROCESS BETTER 
'~ lioRE WIDEI^ L)NDEfeST0OD AND TO REGUIJOIIZE THE 

• . - THE SCIENTIFIC AND'lAY CO^flUNITIES COWTRIBOTE INFORMATION IN- 

l^roSTS, TOE Ni4hOULD publish PERIODICALLY (e.g. , ANNUALLY) 
If SuN^ESblmiS cL^ ATI^ON TO 1«E. 
■ ' S^IED WITH WE GRA^ -PEER REVIEW SYSTEM .A^ WHICH . • > 
• iNVITES' sUQGESnONS RBGARDINGTANDIDATES FOR SPECIFIC IRGS. 

The announcement should, appear in the Federal Rf?^^^^'^. 
NIH Guide to Grants and-Contracts, and brief notic^ should be. 
sent to^ientific journals, such as Science, etc. Also, the 
.annouSSnent should be circulated' amoPiTtK^e individuals, , 
. institutions', scientific soc^et^ies, and othe? organizations 
i^hich request that their names be maintained on a mailing list 

• ' for this purpose. , « • 

The annou^Krement should include the following items of information^ 
about: . . < 

(a> the NIH grants peer;revie\jf system: 

. o a. brief statement of the role and' importance ot m iPG^in the 
"^peer review process; 

o a description of the nature and' scope of an IRC? member's duties/ 
" responsibilities, and work-load; v. ' • » 

o a description of the key steps in the reviewer selecbion.. ^ 
process; - ' ^ 
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o a listing of the pifefesslonal Characteristics that NTH looks 
, - for in selfecting IRG mentoers; and ' ' 

^ ^ p a discussion of the other factors v*ich*are talcen into 

consideration in^electing^ tRG members (e.g., geographical 
and institutiaiai spread, r^resentation of ethnic minorities 
and wonen, arid other committee management limitations), 

,(b) eacdi specific 1R3: * 

t , its function; ^ » * : 

1 o the number of vacancies to be filled; 

o the areas of scientific and, technical expertise to be 
represented' by the newmlarbers. 

Reference to the regulajrly published roster of NIH Public Advisory 
• Groups should be included. (1) * ^ ^ ' 

It should be stated that potential respondents must document the 
qualifications of each individual suggested t attachment of -curriculum 
vita^ and bibliography, etc. ) and specify the IRGjs) for v*iich the 
individual is to be con^sidered, if suggestions' are to be considered. 
There also should be a clear statenent that NIH, ^ile willing to - 
ccMisider all submissions, reserves the right to make final selecticiis 
^ without presenting public justifications or accounting specifically 
~ either to those who suggest candidates or to those who are suggested. 

/ It is further recommended that responsibility for coordinating the 
preparation and .distibution of the announcements, and for providing 
^ggtssted names to the appropriate IPG Executive Secretary should 
be centralized within the Office of the Director, NIH. 

In addition, in order to assess its impact, the effects of th^.s new 
procedure should be evaluated ^ carefully selected criteria within 
a given period of ^ime (eig. , five ^ears after implemaitation) . 

2. ' THAT, IN ORDERS PRCVlDE FOR AE»ITIONAL ADVIC^ BEYOND THAT 

GIVEN, BY INITIAL REVIEW GROUP MEMBERS, PARTICULARLY IN REGAI?) TO ^ 
- THOSE RESEARCH GRANT APPLICATIONS IN WHICH NEW SCIEWTIFIC FIELDS ^ 
ARE TO BE EXPIRED OR UNIQUE PROCEDURE ARE TO BE DEVELOPED, THE 
NIH SHOULD IMPLEMENT A FORMAL PROCEDURE WHEREBY AN APPLICANT MAY- 
IDENTIFY THOSE PARTICULAR ASPECTS OF THE PROTOSEd RESEARCH WHl6i 
ARE CONSIDERED TO BE UNIC3UE. AND TO SUGGEST POSSIBLE CONSULTANTS 
^ :WHO ARE CONSIDERED TO 6E LEADERS IN AI© CURRQiTLY KNOWLEDGEABLE 
/ AttWT HIS/HER AREA OF RESEARCH.' HOWEVER, THE DECISION AS TD 
. - WHETHER TD SEEK AEVICE^FROM THE SUGGESTED EXPERT(S) C» ANY OTHER 
EXPERr(S) MUST REMAIN THAT OF THE NIH. 

(It is suggested tehat sudi proposals Should be made in a separate 
letter which accarpanies the grant application.) » - 



ERLC 



82 



3. THftT, THE CURRENT PRACTICE OF CONSULTATION WITH PRESENT, , 
' 'retiring^ and former IRG I®BERS CONCERNING POSSIBLE NEW MEMBERS 

MOULD CONTINUE to BE ^ ONE OP A SERIES OF STEPS BY WHICH 
ADVICE CONCERNING POTENTIAL IRG ME^fflERS IS SOUGHT FPOW MANY 
j SOURCES. ■ ^ - " . 

4, THAT THE CURRENT PRACTICE^ OF PROVIDING OtffjY ONE- NOMINATION 
FOR EACH IR£? VACANCY SHOULD BE RETAINED. FUPOHEHMORE, ADVICE 
REGARDING SUCH NOMINATIONS. TO DRG INITIAL REVIEW QfOUPS SHOULD BE 
SOUGHT FROM KNOWLEDGEABLE AND INTERESTED STAFFS OF THE VARIOUS BIDS 
RESPC»ISIBI£ FOR THE ADMINISTRATION OF GRANTS IN THE PARTICULAR 

• BIOMEDICAL SCIENTIFIC AREAS* • . 

The above recarmoidat ions', when inplanented, should lead to the. 
selection" of IRG members in the future by an open, objective process 
whidi assures, that, as, in the past, only qualified persons participate 
in peer review. • ' . 



SELECTION OF INITIAL REVIEW GRDUP MEMBERS 
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SELEnraOH OP MEMaSRS Ot SPECIAIi INltlMi REVIEW GROUPS (SIBSs) , 



Generally, tesearch graht appljlcaticxis are' reviewed by duly constituted 
^Initial Review Groi?)S (IRGs). However, certain conditions described^ 
4<felow) precliKle the use of IRGs for the review of special V/pes , 
of researcii grtot applitations. In such cases. Special Initial 
Review Groiqps (SIRGs) ar^, organized, tii^ inembership of whith reflects . 
the review needs oS a particular rese:^rc4l grant ♦applic5iti<bn qr group of 
applicatiOTs/ ' r . ^ 

U CURRENT PR ACTICE - SPECIAL INITIAL REVIEW GROOPS ' - 

- . ^ : ( ^ ^ ^ , 

Uie selection Qf spekrial review group ^consultants is based on the guide- 
• lines set forth f or" IRGs (1) cortsjdering. major factors such as scientific 

^ ccrapetenceV and expertise in ^ particular scientific field as well as - 
as standing in the scientific conmunity and o^er factors sudi as condi- 
tions which could create conflict of interest situations, r presentation 
of women and minority g'rou|>s, and the geogra{*iic distributi%i of qc^uI--^ 
t^nts. ^ , ■ * 

1. Conditions jwhich govern the assignment of a research gi^ant . , 
application to an SIRG includef ; , , 1 

^ »' . - _ — 

* a. SuhmissiOT by menbers of ipGs of applications for whidi there ^ 
1^ is n§ other JIG with the appropriate expertise to undertake 

review. ' 

^ b. Sutmissic^ of appllcatiOTS of sudi conplexity that the scien- 
r • tific content overle?)s the . review area of two or more IRGs; 

e.g.. Program Rejects,. Research Centers, Clinical^ Centers, or 
Biotechnology Resdufc? applications. y 

c. Submission of applications in response to BID announcenents of 
specialized program interests; e.g.. Research Fellowship Awards, 
Request for Applications (RFAs) , of International Fellowshijp.^ 

d. Submission of applications for which it is determined that there 
.,are other special review requirements. 

2* Process of selection of SIRG members 

The SIRG Executive Secretary reviews the application(s) in question 
and develops a roster "of pot^ntia^l reviewers after consultation with 
the Program Director of the'BlD to v*iidi the application h|is beea 
assigned, making use of his/her previous experience and tedinical ' 
. - insight, and that of other Executive Secretaries having coraparable 

review «cperience*. , ' a * , 



other contributing source^ frequently used to obtatin consultants 
. for an SIRS include: reviewers suggested by the Principal.In^festi- 
gator of the application to be reviewed, individuals identified 
in reference citaticxis in the applicatiwi, reviewers who have _ 

• conpleted tenyre on IRGs, as well as, on occasion, current IR? 
.members v*io may haye unique eagpertise. tihenevei; possible^ftighly 
qualified young investigators are invited to serve along with ' . , 

. their more senior -cobnterparts. ■ 

' *• ■ ' . ' • . 7 . ^» 

/An SIHG generally consists of^a minimum of 5 and a Maximum of 1^. » 
depending upon the- size* and coiplexity of an appllciition 'is well.; 
as the number of. similar proposals to be r^iewed. If a project 

• sit^ visit is required for the review of an applicaticxi, the Pri|t- 
cipal Investigator receives a copy of the SIRS rdst^r together .with 
other pertinent details relev&nt to the pendii)g site visit., Hie 

. final SIRC^ roster may be modified upon the^requestvof the Principal ' 
Investigator. .However, assignment of an complication to an SIHG 
does not always insute that a project site visit ^ill be made» , ^ 

Upon conpletion of a specific review assiAment, the SIHG is dissolved. 

IMPMir OF tfiGlSLATIVE PROPOSALS ^ ' 

V 

'k threshold question involves what cbc^titutes a "canmittee." It is clear 
from the above that there is a recurring need to call togethelr groups ' 
such as SIRGs, and, v^ile the Federal Advisory Committee Act (FACA) does' not 
distinguish between short-temhand more permanent advisory ^groups, ^evidence 
that has "been presented in this report would allow a strong argument to be 
made ti^t "special" groups formed to review a single application 'or group 
of applications are not committees for the purposes of the FACA. however, 
pending legislation (Senate -Bill N^. S 2947) would amend the *def initlon 
ofN"advispry committee" to enconpass "ad* hoc" groups thus making it ' 
impossible to set up SIRGs needed to provT3e for the important situatiorte 
described above. . " ■ : . / 

RECOWIENDATiaiS AMD DISCUSSION 

' i., THAT IT IS ESSENTTAL .THAT NIH CONTINUE TO HAVE TOE FLEXIBILITY ^ 
^ AND OPPORTUNITY T9 ESTABLISH SPJSCIAL. INITIAL REVIEW GROUPS. 

2. THAT UlH AND HEW SHOULD OPPOSE LBGISLAJnON REGARDING Jto HX GH3UPS. 

Ihe review of s^plications assigned to SIRGs. is conplex. The work.- ' . 
load is uneven and inconsistent. At times there may be a large 
number of applications to be reviewed; at other tiroes there may be 
only a few; but in almost all cases the applications are conplex artd 
extraordinary skills are required in organizing appropriate SIRSs.* 
Timing is iirportant depending on ejvailability of consultants willing 
to undertake review assigranfents v*iich may require an absence of^o to 
three days away f ran their own: institutibns> frequently on short', * 
advance notice. Furthermore; a mecfiainism for the proper review of 



applications fron IPG merjbers is necessary. It is therefore essential 
that) NI%continufe" to naye the flexibility and opfwrtunity to establish^ 
'SIRGs freed fjcom th^ formality -and delay involved in establishing^ IRGs. 

. ° ' ' ■ • ' > • • ■ , ' .f 
-3. H^^T THfe BDSTEI^ OF CONSULTANTS TO BE INC^JMID <SA A SPECIAL. 
INITIAL REVIEW GPDUP BE SUBMITTED TO- ALL INVESTIQATORS . 



Under the present system only those principal investigators for whose 
. projects a project^ s^e visit is plarir}^ receive the roster of.consulr 
tants prior to the -site visit. Iti is -proposal would provide equity* for 
all investigators- whose applications, are reviewed by SIRGs. 
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i NOMINATIOM, SEEECTION, AND :ftPPOINTMfeWr OF ADVISORY CXXJWCIL JCMBEBS 

Each «f the aw^ding Bureaus/ Institutes, and Divisions (BIDs), has a ^ 
Natioaal AdvisSy Council or Board to offet *advice'and.make reconmendation*^ 
on niattefs relating to the mission and goals of the Institute- it serves, 
•me Councils and Boards are specifically charged to review applicatiorts for 
.grants-in-aid relating to research projects, gtants and awards for research 
training and other activities as may b6 defined for acconplishing the 
mission of tfie'awardiiig BID. The Council • receives reccnmendations on C . 
. i^jplicatioris tor grants from initial peview groups, which have been chargeH. 
with ^making' the scientific and te^^cal merit review. Ihe Councils or , 
Boards advise on policy matters relajfced to th§^development and management 
~ of grant programs and providfe final ,review for -scientific merit and program 
relevance of ^lic^tions for grants-in-aid. "niey provide periodic review, 
' and eveduation of grant progreun ac<?birpliSttnents, help to identify and 
resdlve prCAjlems in program devel?»ment^ and reconmend guidance for th« 
Institutes in the context of natic^nal" health prograns. , » • 

•As established by laW, each Counc'il and Board is advisory to the Secretary, 
CHEW, through the Director, NIH, and the Director .of the. awarding-Bip.* 
K)r the most part, the melifcer^ip of each Council or Board consists of the ■ 
Secretary, DHEW, or his aesigneje, plus- leaders in the fundamental* sciences^, 
medical sciences^ education am? public affairs as may be approved by the - 
. Secretary^ * ' 
' . • J • " . ' 

, I." ' PBXEDURES'AND gllDELINES FOR SELECna?N)F COUNCIL OR BQftRD MEMBERS . . 

The Public Health Sgrylce (PHS) Act CD directs the Secretary, DHEW, 
. - to a^spoint menfcers to NIH Advisory Councils in accordance with the ^ 
requirements set forth by each of the sectiorfs of the Act which 

_ spply to the pvticular Council.** These requirements,, in general, ^ 

are designed to provide authorities knowledgeable in the progranmat^' 
areas Within the Institute's mission; faipiliar with the NIH procedures, 
aware of the rol^ of diverse' institutions in bionedical researdi, 
and mindful of thfe health needs of the ;^rican "people. To provide 
this range" of expertise, each Council consists of both, scientists 
and public mentoer's. Ex off icio . members from othei: Federal agericies 

J are appointed as required by law. . ' 



*witfi the exdeption of " the National Ca^ncer .Ljvisory Bcjard which is 
advisory directly to the Director, NCI. • \^ 

**Moi*)ers of the Board of Regents of the National Library of Medicine i 
and of the Naticwial Cancer Advisory Board are appointed by €he President. ^ 
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Criteria for Selection 
, 1. Scientific Members 



4 



Individuals selected as s^ientif iojnenbers are chosen, from among . 

. those reScJognizetJ as^outstaridiiki^ lea&ers< in* their fields and 
considered ta have a brtp^ inter^^ in^nd understanding \)f the 
relaticMTShip of bicmecjical science fee? societal needs. It is > 

, expected that the Goi^Acil iReinbers will 'have recognized achievement 
in a relevant fieldi' other sci*entifio\^cccxnplishments, ^nd honors. 
Itie' assessment of %pt0priiij:e aJ^ilAties and aqcoirplishraents is 
based jirimarily on .the gen^tajf xeo6gn^tioji^ by peers in ^e bid- 

* medical areas as a broadly based expert m thfe current and past 
development and status of a, particular fieM of research. In 
addftiiDn to superior JaGcomplishment, the characterdstici of ^ " 
mature jixJgment, balanced j>erspective, apd darjectiv^ty are 

•essential* -'Candidates must, have demonstrated the capability 

, of working effectively in a group context as-cromnittee nMnbers* 
.Ihose who accept appointment are expected to 'accent responsibility 
for all work assignments. • ' ^ * 

2. Public Menbersf \ • ^ - . . \ . ' * > 



Individuals selected as public members should have a demonstrated 
interest ^ background history relevant to the program of the v 
Institute. These individuals should have"lcnowl,edge concerning *the 
need^ and aspirati<xis of society in ||ie areasof th^ missipn c5f . 
the Institute. ^ * ^ ^ » 

3. General Criteria and Policy for Selection of CounciJ^ Members^ 7 

a. Within a given Council,. representation of needed scientific 
, disciplines or medical research specialties must permit the' 
attainfhent and maintenance of a proper balayice^to cover thef ^ ' 
rangp of the mission and goals 6£*tHe Institute.* ' 

.b. Wl^enever possible, selection of candidaftes. sjiould reflect 
an equitable representation of the geograjAiic regions of 
\ the U. S> These considerations-, however, mu^t remain / • - 

^ ' subsidiary to and should not take precedence over the v . " 
appoirttment of individuals of the highest qualifications. 

c^-No two members of a'Council /should be appointed f rqm the . * 
same institution or organization. * . * ^ 



**lihese general, criteria arid policies apply only to adyiiisory bodies appointed 
by officials within HffiW. Even where the appointing off icial is' the . ^ . 
Secr§tary*or other DHEW official, exceptions to these criteria and pojicies 
lmay be permitted. , \ U 
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>Io**new rtentoer shpulcl be appointed to the,«ajne or apy other 
' . ' . CoC»hcil witiiin a year gfter tfermin^ion of a prior , 
' ' ' . . fippointm^t . ' No matter -s.erve *concftiy:ently on^ mdre 
than one DHEW -advisory gK^, 

. / e. Within .the criteria, gui^Selines, and ccxiditions noted f 

* abovfe, NIH pf active is to. utilize the widest available 

• pool of qualified candidates for advisory service, - 
. • ifKiudjjig' ec^itable' representation of minorities; w 

.1 ^ ' . : and yogth. Considering -all these factors, primary consider- i 
• ' ' ation for Codicil nofiinaticwis should be giiv^n to scientific 
'// leaders in thglr specif ic 'fields' of competence and. lay ' 

' . ' r> representatives of demonstrated .interest and background 
. relevant to* the program of the Institute to vrtiich they 
^ " • ' . , have bpen nmifiateS ' as <^ouncil members, . ' * * 

' ■ • ]^ ' '4/. Pgbcedufes for^W^ Advisory Couxxc^ ^j^j^ 

^e general procedure is' for NIH to nominate one priro«yar5^ 
: . ^ ' one alternate for each vacancy on -an Advisory Council. - (5;ach ^ 

professional br scientific vacancy on a Council -is usually 
.limiti^ to a specific medical or scienti|jLc need; see.A-3a.) 
. llie Office of ^the Secretary, E«EW, maif request aBditional . 
nominaticMis and/or roaHe nomination^ , 'itself • t^e the S^retary, 
' * " ^^DHEW, selects a candidate to ^f ill a QounciJ vadSuicy, he sends 

a letter of invitation. Itie PHS Afct ditects the Secretary to 
* ; appoint monbers to Advisory Councils in accordance with the 

reqfuiranents set forth by each of the s^ections of the Act 
• ^ich apply' to the particular Councid.. A recent and most . " 
welc€me addition is that -all' appointments to advisory 
coitmittee6 to afesist in impl^nenting the Pubic Health Servicfe 
^-Act are required to be made without regard to political ' . > 
affiliation.* ' . - ' 

H;. DEFICIENCES AND/OR PROBLEMS ENOOUNTEBED IN CURRENT SELECTIO N PRACTICES 

* ' ' ' ' I 

' Apppintments to Council? are f or, oveclappin^ terms of four years. _^ 
^ Vacancies occur automatically at th6 expiration date.- Unles's fiHeq 
^ in a timely fashion, a.vaciauiri is created and the",Council will function •, 
at less than opt imuro efficiency. Maintaining a Council at full strength 
is essential if it is to function effectively. At present, the 
/ . has no Rhowledge of or control over the timing of appointments of new 
. Council.members. . Both de^^ed appointmer^ts and 'unfilled vacancies 
continue to threaten the efficient (^ration of. the Councils. For 
example, there are 153. authorized Council positions. One-qliarter of 
. * these^terminate aflhually as indA^idual terms expir^. Forty-four . 
'appointments are needed to fill the vacancies for the calendar yeatf, . 
1976- As of Noverrtieij i; 1976, the 44 positions still remain vacaqj:. 

• • • . ' 

♦section lOOl, Public Law 94-278/Health Research, and, Health Service ' ' 
Aiaandroents of 1976: ' • 
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Uie PHS Act ^authorizes the ^Council to advise on policy matter^ related 
^ 'to the development and man^ianent t>f grant programs* It also authorizes 
^ ' the Council to provide a second review for scientific mesit and prqgram 
relev^uice of application's. 'for grants-in-§id. In general, no grant can* , 
be awarded unless tecaqmended for approval^ by :the awarding unit's advisory 
J council or board. Seribas/c^rational problems can and do ciccur 
- • wheo substitutes for NIH.l?k|ninatec3 candidates arfe ma^ without NIH 

knowledge and concurrence. In such instances, the ^f^crete balance • 
of scientific specialities, necjfessary to coven the range of areas of 
expe||ise required by the Institute's programs can be destroyed with 
the resultant -effect that the Advisory Cfouncil may have no represent 
itiQn for one* or more researqh areas jand other areas may have • 
>licate r^esentaticxi) • Unless there is ddecjuate*arici appropriate 
representation 'for each of the major areas of COTcern to each * • 

Instltpt^, the Gouncil will lack the ability to'carryjcxjt its 
; statutof'y ^responsibility with maximum effect ivehess; * 

ill. RtX3yME^^D^TI0Ng and discussion -/ 

1. THAT. NIH SHOULD SEE? TO HAVE THE AUHOILTY FOR SELECTION AND 
APEOIMIMENT OF MEJ^BERS OF ADVISCKY COUNCILS/BOARDS DELSSATED^ 
TO THE ASSISTANT SECRETARY FOR HEALTO, DHEW. 

Maintain'ing a Council at full strength is , essential for ,the Council 
' to effectively meet iti' responsibilities a^ a reviewing body for 
applications and advising the Instijbute on program priorities and 
y balance.^ At present the NIH has no knowledge of or control over the 
selection or appointment of new Cc>unoii mejnbers. Unfilled -vacancies, 
delayed ^fipointments, and the appoirftrooit of individuals lacking in 
the required expertise, continue to thre'aten the effectiive operation 
^f .Advisory Councils. It is essential that sonjecme knowledge^le 
ot the needs of the Institutes, and the roles of and requirertlRits 
for Advisory Council mentoers, has responsibility .for assuring that * 
V the ^otal selection and appointthent process proceeds -in an effective 

and timely maimer. Ihe Assistant Secretary for Health, th$<highest 
department health official, has the Background and scientif ic/knowl- • 
edge to carry out these requirenents. . • 

2 . raAT" THE PROCEDURE FOR NOMINATING AND SELECTING COUNCIL MEMBERS 
BE MORE OPEN. SCHEDULED COUNCIL VACMICIES SHOUIX) BE ANNOUNCE 
AND PUBLISHED EARL^ IN THE -FISCAL YEAR. THE ANNOUNCEMENT SHpULD 
SPECIFY yne COUNCIL. INVOLVED, WEI CRJIBRIA FOR TOE SELECTION OF 
CdUNCIL MEMBER?, THEIR DUTIES, AND TERMS OF APPOINUffiNT. , IHE 
NAMES OF INDIVIDUALS SELECTED>SHOULD BE PUBLISHED TOGETHER WIT^ 

■ HffilR C3UALIFICATI0NS. ' J 

I . ^ 

Hie Grants Peer Review Study Team, in assessing the perc^ions of the 
scientific and.. inform^ lay ccninunity as to the duties of.and criteria " 
for selection of Council menbers, became aware that there was -misinfor- 
mation and, in many cases, lack of knowledge as to the duties and 
functions of the Advisory Councils as -wtll as the criteria and methods 
for <tibe selection of Council, members . Accordingly, the QPRST urges 
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that the?e be wide dissemination of information aA to the duties and 
functions of the Advisory Councils and the methods by v*iich members 
, are selected, . . * 

3. THAT, WHEN A SELECTION HftS BEEN MADE FOR A OCXJNCIL VACANCY{IES) 
Om^R IHAlJ FBO^ NOMINATIONS) SUBMITTED BY THE DIRECTa|r NIH, ISffi 

— APPOINTMENT SHOULD NOT BE MMJE FINAL UKTIL THE DIRECTOR, NIH, 
HAS' HAD THE OPPtpRTONITY TO CCJJffiNT CW ,THE SELECTIGN. 

Through its experience, the NIH has acquired considerable knowledge • 
orthe apprc^r lateness of n»any individuals to 'serve on a Council and 
afrapprefciation of their effectiveness. By inpleme^ting the above 
proceffure, the^risk of appointing inappropriate "or" potentially - 
ineffective menbers would ^be avoided. , ■ . * ~ 
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CONSIDERATIONS IH REXSARD TO 
CONFLICT OF I WTEREST 

— — • ■ — ■ — ' , : > 

OHE UTILIZATION' IN NIH COMMITTEE MANAGEMENT , 
OF THE FCm HEW 474 (CONFIDENTIAL STATEMENT ' 
OF EMPLOYMENT AND FINANCIAL lOTERESTS) 

' ' ' ■ > 

In accordance with Civil Service Conmissio'n and HEW requirements, NiH 
periodically secures from' each member of its advisory committees a,Fornl 
IffiW 474 1, .regarding the nienilDer's eraployinent and relevant fj^nanqial interests. 

BadcgrouiKl ;^ . ^ • , 

Qfener ally* speaking r the Federal conflict-of-interest statutes make it a 
crime for a specif Government employee (such as an advisory comdttee 
nien45er v*io is* not otherwise employed by the; Goverranent) , except in the 
'discharge of his/her official duties, to represent anyone else in a 
particular matter in which the United States is a party or has a direct 
and -substantial interest and U) in which he/she has at any time partici- 
pated; perronally and substantially in the course of hi'sAer Gover*inent 
entployroent, or which is pending bgffire the Goverrnnent agency he/she : , ; 
serves* There ''is ^so a criminal sancti^ against an employee particijMtlng 
as part of» lifi;;*ller 'official 3tities in a particular matter in which, 
to his/her knowledge, the* qnployee ' s spouse, minor child, partner, 
or a profit or nonprofit enterprise with which he/she is connected, 
has a financial interest. 1/ {Appendix E-1) 

As written, the conflict-K)f-interest statutes simply^ describe the pro- 
. hibited activities arid the penalties involved, but do not specifically 
requir^m^encies to take steps to prevent potential violations thereof. 
Nevertheless,* although individuals are, in theory, charged with notice 
of the criminal laws, it is often not readily apparent that the laws, 
apply in certain Ccmtexts. -Hence, agencies do undertake to assist 
employees in avoiding violations of these laws. • This is reflected, 
for "example, in the HEW Document, St:andards of Conduct, which discusses . 
conflict of interest" in detail, sets forth a procedure 'f6r obtaining 
adviqe on sudi matters, and even l^as a separate subpart addressed 
specifically to special. Government employees:. Another example is the 
procedure outlined elsev^ere in this report by which NIH deals, with 
applicaticxig from advisory Committee menbers* own institutions or from 
the members themselves. 



1/ It should Wemphasi2ed'that this i^ only a general description, 
of these statutes. For 4 mor^' detailed discussion, see the HEW 
StawJards of Conduct, 45 C.F.R. Part 73. 



II. The Form HEW .474 



Only the individual member normally has the basic information about 
his/her activities which is necessary to avoid all but the most obvious 
conflict-of-interest situations. A major purpose of the Form HEW 474 
^Appendix. E-2) is t,o direct the mCTibet^s attention , to ' the subject of 
conflict of interest and to require hin\/her to consider his/her •own 
activities in light thereof. A second goal of.th^form is to aid 
agencies in icjentifying* situations in which potential conflicts* may 

Agencies are 'required to secure this form from 3pecial Governroent| 
^ employees by Executive Or^er 11222 which reads in pertinent, part as 
follows:. 

"Sec. 306., Each agency shall, at the time of atploy- ' • 

meht of a consultant, advisor, or other special Govern- 
^ ment ^employee reqfilre him' to sufiply it with a statement 
of all other employment. The sta^iafent shall list the 
names of all corporations, comjpanies, ficms. State * 
or local governmental cfrganizations, research ^organi- 
zations and educational or other instrtutic«is*in whidi 
. .he^s serving. as employee, officer? .'member , owner, 
• director, trustee, advise^:, or consultant. In addition, . V - 

it shall list sucl^ other fifiancial information as 
the appointing department or agency shall decide is 
relevant ir^M^ light of 'the duties the^ at3pointee is 
to perform. Wie appointee may, but r»ed not, be required 
to reveAl precise amounts of investments. Ihe statement 
shall be kept current throughout the period during 
whi<^ the emplcqreje is on the Government rolls." 

• • '\ ' * ' ^ ^ . * 

,ltiis pr6vision-has beerl impl«nented by Civil -Service Coninission artd 
' HEW regulations. 2/ (5 C.F.R, ^735.401 et seg.? 45 C.F.R. §73.735- 

1201 et seqv) Among other ^things, these regulations prelsciibe the * 
^ contents^ the Form HEW 474; require that the form be revKyed by a 

high level agency official; ^and specify that the form be maintained in 
' thfe appropriate personnel office but separate from the official personnel 
folder. The regulations permit agencies to make the conpleted forms 
. . available c^y as specifically authorized by the head of the 
operating agency . . . for gopd cause shown,. . but this would rfot - 
^ prevent full-time Federal employees (such as Executive SecreUries) 
from^ looking at the forms in connection with their official 'duties. 

- In order to implement the aforequoted portion of the Executive Order 
at it§ level, NIH has published Manual Issuances 2300-735-1 arid 
^^300-73572, dealing with conflicts of interest on the part of special 

:v^i&™ent employees. ^ The former (Appendix E-3) Is a ofte-page document 
whi<»i pimply directs consultants arid acjvisors to submit Forms HEW 474 

2/ JS!t)e noi regulation are.ccjjwtfnly referred to as the, HEW 
, Standards of Conduct. / 



and to iflp4^e than on a yearly basis and requires NIH statf to issue a 
copy of the; Standards of Conduct to all newly appointed or' reappbinted- 
consultants and advisor s^r together with a roeniorandum from the Seqret&ry, 
HEW, alertingMffiW personnel to .become familiar with the contents thereof. 

As had alr<e2K3y be^n indicated r the Standards of Conqluct regulations 
require that each Form HEW 474 be reviewed by a'high l0vel official 
"... to determine whether conflicts of interest Or apparent cwiflicts 
might arise from the acitivitie? reported thereon J Ihe Standards^ also . 
contain sections relating to such, specific matters as professicmal 
and consultative services, writing and editing, publishing, teaching, 
and holding office ij^ professional societies or. State or local government. 

NIH Manual Issuance 2300-735^2 is entitled "Conflict ot Interest^ 
Policy for Committee Mentoers." (Appendix; E-4) . It sets forth guidelines 
for the selection and appointment of members to avoid 'certain types oj 
conflicts r but places on the coriimittee member . . the primary responsi- 
bility for evaluating hi^ financial interests or those of Ms family, 
that relate directly or indir^ly to his duties." It assigns to thfe ' 
BID Directors responsibility for determining who shcQl review information 
oi> the Form HEW 474 antj \^o shall have access to such iiiformatiofi. The 
only specific responsibility given. to^the Executivl^ Secretary is to remind 
oonmittee menters periodically §bout . . l*IH and other pertinent policy 
regarding ^oidance x>f 9onflict-of-intey est situations." 

Since the Executive Secretary plays such a qentral role in the actual 
operation of coritaittees, the Director,. DRG, has established a policy 
(Appendix E-5) for the ORG ftudy sections which provides as follows:, 

"1. Bach Executive Secretary prior to a regular * « 
study section session, ^^hall re\Fiew. in (Mrs . Pollak'sJ 
office the forms 474 on /file for that stud|p . ^ • 

section'. Such notes as^ neosssary may be made. 



2. Study section miembers \k)d identify tjtiemselves on 
Form 474 as affiliated with 6rganizatiOns shall 

' not participate ip the discussion or vote 
an, a^lication from'any organisation soUiited.^ 
/ Ih^other words, the^ present ^policy of nonpprti- 
\ cipation'on applicfitions from major employers 
shall' be in effect fc^ these organi?2^ior^. 

3. At ^^^he close, of each /study Section 
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'meira^r must certify 
the discussion of majbte^rs from any 
or institutional sysftems with which 
is affiliated. ^ The 
to be signed will be 



Jiat he/she was 



^xact wording- ol 
developed fay 



^ Scientific Review Bi anch and approvj 



As you will note in 
practice for counci 



the report, thif 
onemfaers. 



tind, each 
Tit during 
'all institution 
e/she 

the -certification 
e Chief bf the 
by me. ^ • 
oar rent 
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4. ^ Eadi Executive Secretary must prepare a list of 
applications for wtiiqh he/%he excused certain sbady 
section roenibers.from review.^ This list should gjtve the 
; naime of each excused member 'under the pertinent ; 
'applicatioi> number,. This list is to 'be attached f 
to the original copy of the formal minutes aijd " ' ^ * 

retained in the ^tudy section's official files. 
The duplicated cc^ies of rhe minutes shoyld ecmtain ^ * \ 
• a statenent to this ^effect. The standard statement 
* . ' is* to be develqsed -by the Chief, Scientific Review 
' . Branch, for to use,** - ' • 

^ »• 1 * * • 

Ttie Grants Peer Review Studyl TecDnn recognizes that it is essentially 
impossible for an *agency such as NIH to t^e cognizance of sufficient 
information about advisory conmitteev members to note' apd avoid all 
potenti^ conflicts. of interest. Hence, NIH /justifiiibly places primary 
responsibility for avoiding these conflicts €n the; members themselves. 
Nevertheless, NIH also has a duty to see that members are informed ^ 
about conflict-^f -interest considerations pnd to take reasonable st^ 
to^help in the identification and^ avoidance, of potential conflicts.*. 
Hie Form HEW 474 'serves the multiple purpfese of brirgisxi the iconflict- 
of-interest'.requirenents to monbeils' attention, causing tHial^ tq think 
,^bout thes^ requirements in relation to ^eir own situations, an3l*of 
providing NIH with added information for use in identifying possible 
prcA)l€ms in this area, ' \ 

Although tJie^Form HEW 474 contains useful instructions, it is the 
Study Tean's view that they are- too general ta fully meet NIH's needs. 
As a result, some IRG members interpret the' instructions too narrowly 
and fail to list 'relevant information, 

Ihe Study Team endorses the. approach being taken by DRG of requiring 
Executive- Secretaries to become familiar with information on their . 
members' Form HEW 474s. The Executive Secretary is the NIH staff 
person v*io is most knowledgeable about the backgrounds of the members 
and the business of the particular connittee, and therefore is in the 
b^st position to anticipate and prevent potential conflicts. 

Recommendations - ^ * * ^ 

Regarding Form 474: w ^ 

1. IHAT NIH 'DEVELOP DCTAILPCP SUI^LEMEOTAl INSTHJCTIONS TO BE SENT TO 
. MEMBERS ALONG WITH THE FORM HEW'474, EMPHASIZING THAT EMPLOYMENT , 
INCLUDES, FOR EXAf^PLE, FOREIGN- EMPLOYMENT, SUMMER AND PART-TIME 
EiiU»LOYMENr, ME^BERSHIP ON' ADVISORY BOARDS OF ORGANIZATIONS, AND 
. CONSULXANT APPOINTMENTS; AUD THAT RELE\«OT FINANCIAL INTERESIS ' 
INCLUDE SUCH THINGS AS ROYALTY AGREEMENTS WITH OR STOCK OWNERSHIP 
IN DOMESTIC OR FOREIGN' BIOMEDICAL RESEARCH ORGANIZATIONS, COMPANIES, 
ENGAtSED PRIMARILY IN PROVIDING SERVICE TO SUCH OHGANIZMnONS , PHAR- 
MACEUTICAL COMPANIES, AND OTIHER COMPANIES INVOLVED IN'tRUG FiESEARCHj 



2. THAT Niri ADOPT A PBDCElDURE UNDER WHXC^-;POPM,HEW 474 IS RETURNED TO 
•, A MEMBER AS INCOMPLETE^ WHERE SUCH MEMBER MAKES NO ENTRY IN 
•• -A SECTION CF.THE FOm BUT DOES NOT WRITE "NONE" OR SOME EQUl- > 
VALENT m IHE SECTION'. 

' 3. THAT ,THE. EXECUTIVE SECRETARIES* OF I}«3S BE GIVEN ACCESS TO THE . 
. FORMS HEW 474 OF MEMBERS, AS NEEIM). 

\^ THAT REVIEW OF EpRMS HEW 474, AT LEAST ANNUALLY,. BY ^"EflCH .EXECUTIVE 
SECRETARY Bp MANDATED THBOOGHOOT NIH* 

5. .'THAT ALL EXECUTIVE SECRETARIES BE" REQUIRED ^PERIODICALLY .TO ATTEND _ . 
TRAINING SESSIONS ON EVALUATION OF CONFLICT-OF-INTEREST SITUATlfelfe. 

In>^feations to Manbers , ' , 

Under current practice, onco an individual is Selected to serve' on an > 
.advisory committee, if the appointing official is at NIH, the Form HEW^ 
y^4 will^ be seat out along with the invitaticwi to membership. VJbere a " 
/higher level official makes the appointment the Form HEW 474 often is 

not" mailed to the selected individual until after the inv^irtatiort is- made. 

While the format of the invitation letters may vary, in genetal, they 
do not make it explicit that the iWitation^is a continq^t oi)e, and 
that Mnal appointmeot depends on, mooq other things, review pt the ^ 
completted Form HEW 474 to determine whether r 'in light of conflict-of- 
interest restrictions, the individual will be able to participate 
sufficiently in the activities of the committee to mak6 a meaningful 
contributibn to the work of the coninittee. Moreover, in most instances, 
<xice the invitation" is madp, the individual will not receive another 
formal ccmnunicatioTi indicating that his/her appointment is filial. It 
is not surorisinq, therefore, that iJidividuals would normally assume 
. that the mitial invitation is tantamount to a final appointment. 

•fliis assun5>tion is reinforced by the fact that; NIH has somistimes ^ 
included an invitee's name on mfenbfership lists released to the hews 
media and other mendD^rs of tbe public, after the invitee h^s agreed 
to serve, but before the individual's Form HEW 474 has been reviewed 
•fiton a conflict-of-interest ^standpoint. 

In the opiriioh^of the Study Team, this approach dowrplays the iinportance 
^ of the review^ of the Form HEW 474 to an unwarranted exten^ so 'tliat once 
'an individual receives an initial invitation and accepts, at will be 
diff i(»ult for the appointing official finally to reject the Individual 
cm conflict-of-interest grounds, as well as enibarrassing for the indivi- 
dual if such a. rejectibn should occur. . 
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REOOItlEmATIOMS / . 

A. Regarding Znvitlitions to Members: ■ > 

1. THAT ALL INITIAL 'dWiTATIO^JS TO SERVE ON' ADVISOKC COMMITTEES I^AKE 
IT EXPLICIT 1HAT PINAL -APPOINIMEUT IS CCNTUKENT UPON 'REVIEW OP 

/ THE COMPLETEDi POlW HEW 474 FOR COK^lCTS OF INTEREST, OR POTENTIAL 
CONFljCTS OP INIEREST, AND THAT TO^ m/i ME3ffiERaE ECSWALLY'NOTIPIED 
OF THE APPOINTMENT AFTER THE APPOtNTMENT-PRCiCESS (INCLUDING REVIEW 
OF THE 'form HEW 474) HAS BEEN' 'COMPLETED. 

2. 1HAT THE NAMES OF NEW MEMBERS OF MVISORX COMMJTTEES NOT BE 

■ ' -RELEASED TO THE NEWS MEDIA OR OTHER. MEMBERS *0F HJE PUBMC UNTIL 
- a)CH TIME AS FINAL APPOIWMNT HAS OCCURRED AFTER CCJMPLETION' OF 
THE FORMS HEW 474. 
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X^IWMEMT OF BMPIDYEES OF TOR-PROFIT ORGANIZATIONS 
T' TO NIH INITIAL REVIEW GROUPS ' 

Under current NIH poiiqy, full-time arployees pf 'for-profit" organizations 
and 'institutions may not be appoihted to'uational 'Institutes of Health 
initial Review Grpup^,{lRGs). At the- request pf .the Office of the Director, 
the Grants Peer, Review Study Team was asked to review this policy and 
,to formulate a recommendation either as to its retention or for ch^e.. 

^ * " .■ 

I. BACKGROUND * . . - • . * . 

The present NIH poJ|icy vAiich does not permit the appointinent of individuals 
enployed by profit-making organizations to serve on initial review groups 
hatf^its origin in two Presidential documents, issued in February 1962 and 
>ifli»3 by Pr^ident John F. Kennedy. The first, dated February 9, 1962, 
w^^ele^farit strict^ to , "Preventing Conflicts of Interest on the Part of 
Advisors and Consultants to the Government. The secwd^ dated February 26 
1962, was issued as Executive brdet #11007, "Prescribing Regulations for 
the Formation and Use of Advisory Committees. "(2) . Both of these documents 
now. have Ibeen superseded by more recent issuances, the first in Regulations, 
Titie 45,'t>act 73, Standards of Conduct Q), pramilgated tffiW, cKkJ the 
seccnd by* the Federal Advisory CormTttee Act Of 1972 (P.L. 92-463) (4). 

The »sica^tial memofaAjum of February 9, 1962, set down specific ri^gs 
for Wfe appointment of consultants and advisors so, that a determinate 
could be made as to the status of each individual^ in regard ta conflict of 
interest. -The necessity for such rules was precipitated by an opinion 
rendered by the then Attorney General concluding that 'the conflict of 
. interest statutes of the. United States- Code {1& U.^.C. 281) aEplied^dt 
only to cOTsiMtants and advisors when they were actually employed by 
tlfe Gov^ernment, but were also applicable throughout the entire period 
of such 243pointments when enployment by the Government was for forty 
percent or mrh of the total ap{)ointm€jit period. Secticm 281, 18 U.S.C., 
irt general,. precludes a government employee f com' acting in matters that • 
cpme before c|overnment departments or agencies on behalf of a nongovernment 
eiifftpyer frow-vhich he ot she receives compensation. The effect of this 
regulation was to tequire 'NIH to m?ke a*determination ^bout each consultant 
as to %^ether or "not his nprKFederal erployment status created | conflict 
of interest situaticxi v^ile su^sh an individual was acting as a consultant-^ 
to the GoverriiRent'. The present ip* Form 474 is used for this pbrpose 
(Appendix E-2)A 

Based on this Presidential menjprandum, instructions^ weref gxven<:o NIH 
to terminate all .committee manberships asj\of July 30, 1962, and to reappoint 



*Pi file on aU the 'background and histotigaJh documents is available 
< in the Grants Peer Review Study Team office. 
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only those individuals about v*^om it could be determined that there was no 
♦ccmflict of interest. It was found that thfre was considerable difficulty 
in making 'such a determinaticwi in regard to consultants v^o were employees- 
of pro£itHnak^ng' brganizatic»is, and thus a decision was made not to 
rea|9point MSh persons. . » 

Bie pertinent portions of- Executive Order #^.100^ stated that nb advisory 
comnittee might be formed unless it* was specifically authorized by law or ^ 
q)ecifically determined as a. matter of formal record by th4 head of a 
department or agency to be in the public ii^terest. Further, any advisory 
committee # the duration of v*iidi was not fixed by law, was to terminate 
not later tiian two' years f ran its date of formation unless the department 
or agency head specif feally determined that its continued existence was 
in the public inter^t. However , Sectioi 9 of this fixecutive Order stated 
that these, requirements did not apply "to any advisory comftittee composed 
wholly of ^repr^fntatives of <state or local agencies or charitable; 
religioo^/ iedudaticaial, tivic, social welfare, op» other similar nonprofit 
organizaton." Hiis reinforced the NIH decision to exclude eraplqyoes 
frolh, "for-profit" organizations f i;om serving on advisory cortmittees (except , 
on those established by specif iCj^^legislatiOT, e.g., advisory councils). 

As stated previously, 'Executive Order #11007 was superseded by the federal 
, Advisory Committee Act of 1972 (P.L. 92-463). Howeveif, the section of the 
Executive Order quoted above was not included as part of "the new legislation; 
and thus all comtiittees established and utilized by NIH, regardless of 
their composition, are now subject to the requiriements of the Federal 
Advisory Committee Act* Curcient Federal regulations (Title 45, Part 73), 
established under this act, are equally applicable to all consultants 
regardleS^:of the profit status of their non-Federal employer?. 

Thus there must be adequate review of the Cdnf lict of Interest Statements 
\ (HEW Form' 474) filed by all consultants? and/there no longer se«ns to be 
any barrier to, the appointm^it of employees of profit-making organizations « 
to serve on any NIH committee, provided it can be determined that a 
conflict bf interest does not. exist when perfonning the required duties. / 
Indeed, the requirenents in this regard mist be no different for Jbndividaal6 
who are employees of nonprofit organizations. 

II . ' CONSIDERATIO^B BY THE GRANTS PEER REVIEW SIUDY TEAM _ _ ^ 

The Study Team recognizes that ixnpetent scientists are enployed by both 
nonprofit and "for-profit" organizations. Furthermore, it is evident, 
\jBpon review of the Cmflict of Interest Statanents (HEW Form 474) currently 
on file, that many present appointees to NIH JRGs are consultants to 
industrial firms and are not ccxisidered to have a conflict of interest • 
as far as iService on an NIH IPG is concerned. It would thps appecir that 
both consultants to and ^ployees of industry should be considered eqi^ally 
satisfactory for service on NIH initial review groups. , 

The Study Team also is of the opinion that, in some areas tf scientific 
researdh, persons with certain talents, skills, and expertise are to bp 



found in the industrial sector who do not; exist in adeguafte numbers in .% 
the nonprofit ^sector. Hre use of such experts, wten needed in the review- 
of grant applications, fehould notj^e denied to Nlif. Purthermprer in 
recait years, NIH has had tp draw on an ever-increasing number *of* individuals 
tpt coivsultant services irj order to joeet certain other requir«n«its; . 
thus, the pool^Qf scientifically knowledgeable indiVidu^s has to bQ . 
iiKureased. T*ie untapped resourfce of ^.^nployees of the industrial sector * 
would reinforce the pool of available, experts upon which NIH cjn dt^aw for ' 
consultant serjy-ces. ' , » 

Based on the. above caisideratibns and the background \informatic»i, 
the Study Teain makes the following reconfnendations: , ; ' 

III, REObjfgMDATIOBiS ' ^ ^ * 

THAT EMPIOYEES OF "POR^PHDFIT" ORGANIZATKatS BE ELIGIBLE ' ' 
FOR MEMBERSHIP ON ALL INITIAL REVIEW GROUPS CONSIDERING 
GRANT APPLICATIC^^t^iNCLUDING NATIONAL RESEARCH .SERVICE 
AWApD APPLICATIONS) , AND THAT THE SELECTION OF THESE 
INDIVIDUALS BE BASED OH THE SCIENTIFIC PROFICIENCY • . • 
NECESSARY TO FILL THE VARIOUS NEEDS OF SPECIFIC COMMITTEE^. 
THE BASIS FOR SELECTION OF SUCH SCIENTISTS SHALL BE THE . 
SAME AS FOR- THOSE EMPLOYED BY "NON PROFIT" ORGWJIZAITONS. , 

THAT THE CONFLICT OF INTEREST STATEMENTS (HEW FC«M 474) 
- SUBMITTED BY EMPLOYEES OF PROFIT-MAKING ORGANIZATIONS 
nominated' as CONSULTANTS MUST BP REVIEWED BY THE AGENCY 
HEAD, WHO MUST BE SATISFIED THAT EACH NOMINEE CAN SERVE \' , 

' ON THE SPECIFI'C C0^fliTTEE FOR WHICH SUCH PERSON HAS BEEN 
PROPOSED WITHOUT BEING W VIOLATION OF 'iHE CONFLICT OF , 
INTERfeST STATUTES. * . ' . , 

IV. CRITERIA FOR SELKTICTJ OF IRG MERCERS FROM "FOR-PROFIT" QRGANIZATIC^S- 

\ 

The Study Team suggests that the criteiria stated fyelow coAsidered 
as part of the above r;ecommeDdation: ^ ^ ' 

1. > General ^ * 

THE CRITERIA FOR SELECTICy OF I RG MEMBER^ FROM "FOR-PROFIT* 

? Organizations -SHOULD be no different than those for selection 

' Cff" PERSONS from ' "NC^RDFIT ORGANIZATIOB.'*'- ' 

, 2. Specific ' 

a. Recognized achievCTient in a relevant fi^ld . 

For scient;ific merit rwiew of research grant applications/ 
an'T.nd^vidual ^serving on an NIH initial review group should 
have particular competence as an ind^>endertt investigate^ 
in the needed scientific discipline or research specialty. 
Demonstration of such competence is based on the quality 
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of researcj} acc<itpl«hed, product;ivity in.tihe form of 
reeearch reports published iri scienf:if ic joirnals, and * ' 
other significant, scientific activities, accorplishmaits, 

and honors. * ^ , . ^ 

, ' ' I, ^. , • 

Assessment is based p^fimarily on' the. general ^jrecognibiai* 
bjr^ scientist's peers of the individual's abili,ties and 
accomplishmoits as a Garefifl, critical, original, and 
deliberate ihvestigator , and as a broadly teased e»ert^in 
the past develqpment and current adyancement of a particular 
^ 'field of researdi.. Usually the M.D. ,'Ph.D. , or equivalent • 
advanced degree is required; however, a doctojfel degree is' 
not required in those cijrcumstanees v^ere;an iiyJividual's 
experience clearly indicates outstanding ccsnpetence^ ^ 

' Furthermore, a reviewer of applications for Natj.onal^ 
, Research Service Awards must^ have had substantial v-. 
experience in graduate research training, with lin active 
interest in the methods ^nd planning of r'eseqp^ training ' 
in his or her discipline/ 'field, or Specialty, and a/ ^ T 
record of accomplishment in sud^ training. V • - / ^ 

^ . • . * /• • ^ r * . 

b. Mature jud^ent and objectivity as a scientist . 

The characteristics of mature judgment r balanced^ 
perspective and objectivity are essential to the merit, 
review prodess. ' . 

. r ■ 

c* Ability and willingness to serve . 

, (a) A candidate must have denonst rated the capability , ' 
of working effectively in -a group context. 

(b) A candidate v*io accepts appointment to an.NIH 
advisory group is expected to accept responsibility for * 
all work assignments and must protect the conf iiJentiality 
of applications and reviewer c^inioris. 

d. Conflict of interest ./" ^ . 

the Conflict of Interest Statement (HEW -Form 474) submitted 
by a nominee mfust \?e reviewed by the agency head who must 
be satisfied that the individual can perform as a speqial 
• governffinpnt employee without being in violation of ^conflict , 
of interest statutes. ^ 
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V* ^ Inplenientatiori of > the Study Team Recxxwmendeltibn 

In' order tb Implement the above* rea»pendationr guidelines for the 
selectioh of all consultants to initial review groyps, but particularly 
those in rajard to persons from "for-^rof#" organizations r ?hould be 
reviewed jgeripdically; and^ ^ere necessary, hodificatiohs should be made 
80 that the ct iter ia for. selection of individualfii are suich as to retain 
ttte quality/ of consultant adyice that WH expects from its advisors. 
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^ APPoflrtMENT OF EMPLOYEES OF POR-PRDiFIT ORSANIZATICNS. 
TO NIH INITIAL REVIEW GROUPS 

^ferences . . ^ 

.• Kenne(5y, John F., Preventing Conflicts x)f Interest on the Part-of 

Aflvisits and Consultants to the Goverran«it; February 9, 1962, 

^ ' >. . 

. Kennedy'', J<*n F. Prescribing Regulations for^e Formation and • 

Use of Advisory Coraml-ttees. Executive Ordej^fllOO? , February 26, 196J^. 

. Regikations, Fomj HEW-539, Title 45, Part^73, Standards of Conduct, • 
Sepfeenflfer, 1970. , ' ' 

.Put)lic Law 92-463, Federal Advisory Camiittee Act, Octofcer 6, 1972. 
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LEGAL COMSIDERAnONSL RE)GABDING' GBAWTS PE Er' REVIEW 

In recent years* the public, ^ the Congrjsss, the Executive Brandi, and the 
courts' have become increasingly attentive to and supportive of. greater 
public- accountability and openness in Government; Recent legislation, 
court ca^s, anc^ prc^x)$ed legislative action in this area have important 
iraplicationis for the qperation and continued effectiveness of the NIH 
peer review system. At the same time, the Caigress and the President's 
BicroejJical Research^inel^ have strongly endorsed'thig s^^stan,. praising 
its rairnessr c*jectivity, ania high quality. The strengths of. the systeA 
roust be preserved while it is kept publicly accountable. 

I. Background ^ . 

Av Peer' Review LeqislatiOT . - • * 

In discussing the peer review system in a 1973 report, the Senate 
Committee on Labor and Public Welfare endorsed the cpranent that:, 
**The peer review system [at NIH] has given us the best science 
through a* federal agency with the least politicals interference of' 
any governmental process ever developed. It is truly one of the 
great achievements of American goverrtnent.* . . " Under this 
system, each application to NIfl for a research grant is initially 
reviewed for scientific merit by a group of experts in relevant 
biomedical fields. The recommendation of the group is then con- 
' sidere<3 by an appropriate National- Advisory Council or Board, made 
up of both scientists- and lay persons, vAich i^es a final recom- 
mendation to NIH on the application based not only on scientific 
merit but also program priority. . • 

Eadi of the NIH Bureaus, Institutes, and Divisions (B^) awardirtg 
grants has a National Advisory Council or Board whose manbers 
generally are leaders in science, education* or public affairs. 
Section 301 of the Public HealjA Service {PHS)'Act [42 U.S.C. 241], 
the basic authority under vAiich NIH research projf^t grants 
are made, provides that no grant may be awarded for a research 
project unless it^ is first "recommended" for funding jDy the appro- 
priate Nationals Advisory Council or Board. 1/ A similar requirement 
exists in Section 472 of the PHS Act [42 U.S.C. 2891-1], relating 
to National Research Service Awards to individuals ?or research 
* and research training; . In addition, a number of the Councils 
are, assigned responsibilities for* advising their respective 
BIDS on other specific program' activities (Title IV, PIJS Act, 
42. U.S.C. 281, et seg.). . ' . ^ 
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1/ By statute, the National' Cancer Institute and* .the National Heart, 
Lung, and Blood Institute may award research project grants, without 
Council ' or Board review and recommendation, where the awards are 
fbr $35,000 or less in direct 90Sts 
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'in^general, the National Advisory , Councils were established by law*" 
at the same time as .the BIDs t^ which they give advic? and have been 
responsible, by statute, for advising on gf ant applic^i9ns since ^ 

, tbeir establishment. On the -other hand, prior to the ^Bigflpe of 
the National Cancer Act of 1971 [Public Law 92-218] , there «a3 no 
actual statutory requiranent that research gr^t applications' be 
reviewed as well by scientific peer review groups, e.g., NIH Initial 
Be\^iew Groups (IHGs).. However, the Naticmal' Cancer Act of 1971 ' 
aroend^ the PHS Act to require theJ)irecto^, NCI, to . . provide 
^or proper' sc^ntific review of all' researdh grants . . . oyer whidi 
he has author Ry (1) by utilizing to the maximum extent possible, ' 
appropriate peer review groups established within the National 
Institutes of Health . . . and (2) when appr<^riate, by establishing 
• . . other formal peer review groups ^is may be required^" [42 U.S.C. 

. 2863.] ' ' ^ ^ - 

[The National Cancer Act Amendments ot 1972 [Public Law 93-352] in 
effect extended this requirement of scientific peep review to-atl 
" • • . applications . . . for grafts under . . . [the EJIS] Act . , 
tor biomedical and behavioral Research ..." (Appendix F-1). It 
further required that review be caiducted " . . . ta the extent 
practical . ... in a manner consistent with the system/ . . " being 
utilized on the date the amendments became effecti^ ^^nd that not 
more than one-quarter of the manbers of any peer review group m&y 
. be officers or employees of the United States, presumably apart from 
their service as members of the group. i[42 U.S..C. 2891-4.] 

' On March 29. 1976 r a notice, of proposed rulechaking (Ap^ndix 'F-2)' ; 
was published in the Federal Register setting forth ^tJPCgbsed regu-* 
lAiohs to implement the foregoing provisions (41 F'.R." 12986) . In 
general, these regulations codify the pre-existing NIH system for 
scientific peer review of grants. - ^ ' * 

The Health Research and Hectlth Services Amendments of 1976 [Public 
. Law 94-278] include a section which states in pertinent part that 
"All- appointments to advisory conmitfcees established to assist in 
implementing the. Public Health Service Act , . . shall»be made 
without regard to political affiliation.'^ ^ 

Conflict of Interest ' . 

With/egard to conflict of interest /the aforesaid regulatipn^ 
Would essentially codify prior NIH practice which requirei^^that 
men4)erkof peer review groups in no way participate in deliberations 
or actions concerning applications from their own insti tuitions and 
that they/leave the room before any soch discussions. Where one of 
the monbers of the group, v^ich would normally review a peurtioular 
application, J.S actually ncffned in the applicaticm as the principal ^ 
investigator or a principal stciff member, the application is assigned 
to another peer review group with^ the requisite competence, or if 
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none exists, it is assiqncH^ to a special- (ad hog ) ini|tial review 
group, no more than 50 perteent of the memberslup of which may be 
from thjp groups of which tne investiga^tot or staffer is a awnber.^ 

These cictiois have Nt)een^ considered sufficient by the^ D^rtinent of 
Health I Education , and. Welfare to avoid any viol^tim of the ccMiflu^t- 
pf-interret statutes^ and they were -endorsed by the Congressional 
cdimittees which ccmsidered the proposals coinpning peer revietr^ 
that were finally enacted as part of Public Law 93-352. 

' . • ' ... ^ 

The foregoing'p<xiflict.-of-interest. requirements were recently ujAield 
by the United States District Court for the Northern District* of 

.Califorftia, in ah opinion dated February 10^ 1976, stating that 
these requir«nents " . . designed to avoid conflicts of interest 
in the grant application review process (are! adequate and reasonable 
. . . ( Gras'setti v. Vfe i nberger; et al; , C.A. No. C-75-1198-SC. ) 

^ (Appendix F-3). . ^ « 



II. The "Sunsmne" Laws - . 

^ /' ^ ^ : ^ . 

Hie so-called "Sunshine**' Laws affecting NIH are the Freedom of Information 
Act (POIA), the Privacy Act, the Federal < Advisory Corroittee Act, and the 
Goverfinent in the Simshine Act. 

Hie general ' thrust of"th?>reedom pf Informtion Act i^ to jnake available^ 
to the public uppSh request arty and ali'*of tne records^ of Federal agencies. 
• Hie Privacy Act seeks to protect the individual from the Gevernnent^s 
collection and use of data concerrkng the individuals wijthout his or her 
knowledge or consent and to provide the individual access to those Govern- 
ment records which include information pertaining tt> him or her. The 
Federal' Advisory Cgmittee Act seeks to assure public knowledge about 
m^tingsv^ Governmental advisory groups as weil as public access to 
these meetings. Ttie Government in the Sunshine Act, sets forth the^c^rcuro- 
- stances in v^ich NIH may close advisory group meetings to the public. 
* * *" , "* . 

The specif ip provisions of the above Acts inrpact on the peer review 
system because of their applicability to all* significant documents - 
connected With the process and to the (^ration ©f comnittees charged 
with peer "review. Ttie current and potential/ effects of these laws and 
proposed amendments to them are exeonined below in terms of ^^ianges in 
the, peer review system, the 'quality of the peer review ^process, and 
reiLated consequences. 

A. Freedom of Information Act^ * ' 

J. Descriptiai of the Law. 

^Oiis Act (FOIA), origj(nally adopted in 1967 {Public Law 90-23] 
and Amended in 1974 [^>Xlc'Law 93-502], requires ag^Kies to 
make available to the' p^>Iic for inspection and copying, any 
requested Government rec^rds^ (Appendix^ F-4) . However, subject 
tb provisions of the Privi^cy Act, discussed below, requests for 
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records nwiy be denied if tMc^e sought fall within any one of 
nine specified- *'exeRiptions'i set, forth ^n the'POIA. ^Only the ' 
folloidng three exeirptions presently bear, on the peer review 
systeni:' * A -m 

"(4) trade secrets and cpiinercial or financial infonnation 
obtained from a ^)erson ar* privileged or confidential; 

'M5) inter-agency or iritrV-agency menorandvims or letters 
^ich ^Id not be availabBe by law to a party other than 
an agency in litigatic^ witft the agency;*. 

''(SI personnel and medical, liles and similar files the ^ 
disclc^ure of which would coriktitute a clearly tlnwarraneed 
^nvasion of personal privacy.^ ^ • 

The Washington Researdi Project Case. 

Until recently, all re^quests under me FOIA for grant applica- 
tions (funded or unfunded), interim progress reports, suninar;y 
statements of IRS. reviews, and site visit reports were denied, 
based primarily on ex^nption 4. To the extfent that these'>mater 
ials contained discussions of ^the qualifications of ipdividuals 
such discussions were considered to be protected as, well by 
exemption 6. Mditionally, since the summcCry statonents and 
site visit reports reflected the deliberations and opinions of ♦ 
thl* members of advisory groups, and site visitors, these items 
were also viewed as being within exemption 5. 

In Washing ton Research Project, Inc , v, Wein be rger , et alw , 
a publTcTnterest grcnip challenged the Deparfinent's denial of 
the plaintiff's request for the foregoing categories of docu- 
ments, relating to specific funded initial applications and 
funded and unfunded continuation, supplemental, and renewal 
applications. The United States District Court for the Dis- 
trict of Columbia ruled that essentially all the materials in 
question were available under the FOIA., (366 F. Supp. 929.) 
On appeal, the Court of Appeals for the District of Columbia 
agreed with the lower court that the" pro€oools and progress ^ 
reports at ^ssue were not covered by exemption 4, but held 
that the portions of suinnary statements and site visit reports 
reflecting the deliberations of members of advisory groups 
and site visitors were exempt from mandatory disclosure under 
exemption 5. (504 F. 2d 238.) (Agpendix' F-5) . In light of ' 
the appeals court decision, ^^iBiich fs now final*, current Depart- 
ment practice is to release, upon requesMP funded initial' grant 
applications and continuation, supplemental, and renewal appli- 
cations (whether or not funded), as well as interim progres^ 
reports, extept insofar as disclosure of any particular items • 
would adversely affect patent or other valuable rights. In 
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so doing r any ccmfideriticd financial information cdncerning' 
the applicant oc grantee will be dented. As in the past, 
requesits for sunjmfery statements* and site visit r^rts 
largely, be denied, 

•W^vailablity of applications. 

it^ is a^nost impossible in the short run to measure the inpact 
of the roiA and the Washington Research Project decision on the 
peet, review process. It has been suggested, however, that scien- 
tists may be less willing to inclijde new ideas in' their applica- 
tions if they may be available to the pu^>lic, thereby diminishing' 
the usefulness of such an application in SfetMmining which projects 
to fund. 'If applicants do react in thia way^ reviewers will have 
to depend more heavily on site visits, the\ general reputation of 
investigators, and subjective evaluations, Hiis wiU tencJ tp 
work against t^ie new iiiyestiqator and the novel idea. ^ 

w In its Report to th^ President and CongresSr dated Afiil- 30, 1976, 
the President'^ Biomedical Research Panel, established pursuant 
to Pufelic Law 93-35^^ concluded that this was a real cdncern ajn^ 
. reconinended tha^: 

"Tt>e Public Health Service Act • • . should be amended* 
tb provide a statutory exemption from disclosure [underj, 
. . . the Freedom of Inf<OTiation Act for research designs 
and protocols contained in grant applications . . • until< 
the grant^. . . funds have been received>y the ^grantee 
' 'institution* . . . Unfunded grant applications ... should 
remain confidential.** 

In a second Report (June 30, 1976) mandated under the Health 
Research and Health Services Amendments of 1976 [Public Law 
94-278} and entitled '^i^lo^ure of Research Information the 
Panel* reaffirmed its ^^ior conclusion, Expressing the. conviction 
that unless the intellectual property tights of investigators 
are- adequately protected, % . . the federal biomedical and 
behavioral research effort and its impact on the healtti of the 
nation— is likely to be impaired ..." H^e Panel' went 
op .to say that disclosure of yrformaticxi from researdi - 
protocols did rwot appAr to contribute to the improvement of 
.peer review. Rather, the' Panel expressed the belief that 
su<* disclosure . . . could impair the ability of the [peer 
review] system to ensure high-quality federally funded researdi:" 
It pointed out that *• . . . the credibility of peer revj.ew 
would certainly be undermined if it were compromised 
by the submission of derivative propos&ls and applications and 
if the judgments by peer review groups were based cm incomplete ^ 
information." ^ . , 
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^ The f iixSings set forth in tfhe Survey of' the NIH Research Grant ^ 
Peer Review. System, reported elsewhere herein, tend to siippoft 
the conclusions of the Pattel , for only o'ne-eighth^pf the advisory 
' -dbninittee membBrs surveyed favored the public release of all , 
renewal' and continuation applications whether or not they had 
been 'funded. fSupplenent A, p. 55) • • ' 

J ) 

REOOtt^iPftTION t ' 

THAT THE PHS 'ACT BE AMENDED TO PRDVIDE STATOTORY EXEMPriCN. FBDM"^ 
. . REQUIREMENTS Of "'THE FREEDOM OF. INFORMATION' ACT FOR DISCEOSURE OF 
RESEARCH DESIGNS AND PFOTOCOLS PRESENTED IN GRANT APPLICATIONS. 

(This iSr^in essertee, an endorsement of the recomnendation of the 
President's Biomedical Research Panel. ) 

B. Federal Advisory Committee Act (FACA) 

1. Description of .the Law. 

.itiis Act (FACA)^ approved on October 6, 1972 [Public Law 94-463], 
states that "No. advisory ccmnittee shall* meet pr take ^ 
any action until ^ advisory committee charter* has been filed with 
... the head of the aqency to whom . . . [the] advisory committee 
reports . . " (Appendix- F-6) The tertn ^'advisory committee** is 
I defined (swbj^t to certain exemptions not Were relevant) to include 

"... any committeer board, commission, council , conference) 
panel«r task forcer or other similar qroupr or any subcocpmittee or 
other subgroup thereof . . . except that sucii'ierm'excludeig . • 
any camittee which is composed wholly of full-tiipe officers or, 
employees of ^e i^erfil Government. AH' meetings of advisory 

* comnittees must be advertised in the Federal ^ejjister , and 
(subject to tlie discussion which follows) all portfonis must be 
open tb the public. Effective ^until March 1977, there is an ^ 
exception to the open meeting requirementr where "\ . . . th^ 
head of the agency . . . determiffes ^he meeting] is concerned 
with mattejrs listed in Section 552(b) of Title 5, United- States 
Code.'* Section 552 is commortly referred^ to as the Treedoro of 
Information Act,*' and paragraj*» 552(b) thereof sets forth the • 
exemptions from mandatory disclosure v*iich were discussed above 

^ in connection, with the POIA. TtiuSr. until March 1977 r a meeting 
mayr in effect, be closed insofar as the subject-matter being 
discussed r if reduced to writing r would be exempt from mandatory 
disclsoure under the POIA. / - ' * 

Tt)e Government in the Sunshine Act [Public^ Law 94-409] recently 
' ^ *amende<3 the FACA (effective in MarA 1977) to permit an exdsption 
for a portion of a meeting' only where "... the head of tWK/ 
agency . . . determines that such portiop . . . may be closed in 

• accordance with Subsection (c) of Section 552(b) of Title 5, United^ 
States Code.** Section 552(b) was added to tl^ United States Code 
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by Public Law 94-409 and deals primarily with the decision-ntciking 
processes of agencies headed ijycxjllegiaL bcxSies conqposed of two 
or more perscxis appointed Ayr the i>resident With the acSvice and 
consent of the Senate. Subsection (c) of Sectiai 552(b)' contains 
a list of exeniptibns from the requirement that these bodies hold 
their meetings in open session* The subsection provides in pertin- 
ent part that: ^ . ^ 

I 

**Except in a case v^re the agency finds that the public 
interest requires dtfherwise • . [porticHis of an aqen<^ 
meeting may- be closed] where the agency properly determines 
that such portion or porticMis of it^ meeting . • is likely^ 
to— 



"(4) disclose trade secrets and conroercial or financial infor- 
maticm obtained from a person and privileged^ or ccxifidential; ^ 

"(6) disclose information of a personal nature vrfiere disclosure 
would constitu™ a clearly unwarranted invasion of 'personal 
privacy; j 



"(9) disclose information the premature disclosure of v*iicii. 
which would — • ^ • ' ^ 



"(B) ,in the case of any agency, be likely to significantly 
frustrate implanentation of a proposed agency action, except 
that subparagrajAi (B) shall not apf>ly in any instance where 
the agency has already tJisclosed to the public thfe caitent 
or nature of its proposed action, or where the agency is 
required by law to* make such disclosure on its own initiative 
prior to taking final agency action on such propc^al;" 

While exemptions 4 and 6 in subsection (c) of Section 552 (bj^ are 
similar to those in the FOIA, the new law does iiot contain any 
exact counterpart* to POIA exemption 5. On the other hand, it 
does contain a new exemption 9B v*iich could be applicable to 
meetings of peer review groups.' 

lA considering whether to use an exemption. to xrlose a portion of^ 
a meeting under* subsection (c), an agency head must decide if i^ 
is "likely" that exempt matters or information will coiie uRr not 
simply that there is a possibility of it occurring. , 
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that constitutes a "conmittee'*? 



% 



A threshold question concerns what ccmsftitutes a "conlnittee" 
for purposes of the PACA, In order to comply with cpiflict-of- 
interest reqyireinents or to review certain types of TOplicatic»is, 
it is' regularly necefesar^ to call together i*at are qJSltonly 
•referred to as '•l^pecialf " or "ad hoc," initial review groups 
to consider one, or a small number of applications, usually 
vOn a one-time basis. Hie/ FACA itselfi does not. distinguish ' ^ 
between short-term and more permanent advisory- groups rpz does 
it meJce any specific mention of "special," or "ad^^ hoc," groups. 
In Fo od Chemical News, Inc»; v. Davis , 378 F. Supp.Tj548 (D.D.C. 
lS74y, 9 United States District Court held that two s^>su:ate ^ 
informal' meetings between Tteasury Department officials and 
ccmsumer and industry representatives, at which the officials 
secured advice relative ^o regulations or\ labeling' of distilled 
spirits f ^re'cqverediy the FACA^ On the other hand, in Nader 
v., Ba r oody , 396 F. ^jpp: 1231 (D.D.C. 1975), another judge in the 
same^court rul^ that bi-weekly tftiite House meetings with selected 
groups -from th^ private sector were not yithin FACA^s ambit. In 
the latter' case, the court cited with apparent approval* the 
following criteria for determining ^^lether a gathering is an 
"a^isory conmiittee" under FACA: 

"(a) ^Fixed membership, usually selected by a Federal official 
or determined on 'the basis of Federal* Law^ I 

* "Cb) Established by a Federal official or on the baSis pf 
Federal l^w? or, if not federally established, the initiative 

. for its ase as aa advisory body for the Federal Governments^ 
came from a Federal official rather than from a private group; 

**(c) A defined purpose of providing advice regarding a partic- 
ular subject or particular subjects? 

"(d) An -organization structure (e.g., officers) and staffs 

'"(e) Regular or periodic meetings." . 

Under these criteria, a strong argument ccxald be made that "^)eciali 
pr "^ad hoc," groups formed to review, a single iapplication or 
groups oF^applications are hot committees for pu^po^s of FACA. . . 

0 

On February S, 1976, Senate Bill»No. S. 2947 was introduced 
whic*i, among oth^r things^ would amend the definition of "advisory 
committee" to encompass "ad hoc" groups. (Appendix F-7). In light 
of ' the realities regarding turie needed tp charter committees, 
passage of this amendment would probably make it ijiqpossible to 
utilize "special" or/*ad^qc" groups in the amflict-of-interest 
situations discussed aBoyeC^or to meet short teqn workload 
increases in specialized aieas. ' , * 
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, '^^ . ^ 

TSta NIH MJD HEW PRESaW SPECIFIC WRITTEN' OPPOSITION « TO AOT'IiEGIS- 
lATIVE PBCffOSAL EXTENDING IHE FEDERAL ADVISORY COMMITTEE ACT TO 
•AD BX" OR SPECIAL REVIEW GROUPS* BASED ON<THE NEED TO PREVENT 
CONFLICTS OF INTEREST IN' THE PEER REVIEW OF GRANT APPLICATIONS AND 
THE NEED TO RETAIN^ fLEXIBILITY IN MEETING SHOjII' TIME WCSRKIOM) 
INCREASES IN SPECIALIZED AREAS. 

, 3. Closing meetings^ 

^ When the FACA was first passfed, .portions of meetings devoted to 

review of grant applications were closed primarily under' POIA 
exemption%.4 and 6, on the basis that discussions were likely to 
cover matters affecting patent and other valuable lights, and that 
negative catmints about the qualifications and ideas of »investi-; 
gators and others could institute a clearly lytwar ranted invasion 
of their personal pjrivacy. While the deliberaticms of the peer : 
review grot?)S would seem also to be covered by exemption 5, the . ^ 

' Department severely limited use of this exemption, to avoid 

excessive closure of. meetings.. In the Washington ResearcAt Proje^ 
dase, however, Jthe Court of Appeals indicated tha^ the deliberations 
* of the peer review groups, -as set forth^^in the surrroary statements, • 

* deserved ex«nptic»i 5 protection. As a result, the Department 
agreed to a relaxation o& its restrictic»is on using* exemption 5 
to close meetings. Since then, exemptions A^ 5, and 6 have all 
been cited as bases for closure to review grant applications. 

On Aprili 6. 1976> the United States Court of Appeals, for the • 
District of Columbia, in ^i^iqn Ccyisumer Action Project, et al w^, 
v,^ Washburn , et aU , ruled that exemption 5 was available as a 
' basis for closing of meetings, thereby overturning; in effect, 
several lower court opinions ,to the contrary. This decision > 
provided added support for use of that exemption in conhection 
with review of grant applications (Appendix* r-8) . 

As has already been intiicated/ Public -^Law 94-409 carried forward 
. exemptions 4 and' 6, as grounds ftir dqging meetin gs , essentially 
without change. Thus, where applicable, these exemptions can , 
' continue to be cited to support the closure of meetings. On 
the other hand, in its Report on Public Law 94-409, the^ Confer- 
ence Coninittee indicated that one 6t the purposes of the amend- - 
ment to FACA Was to overrule the Washburn decision. At the same 
time the conferees stated that:. 

'*T!tye conferees, however, are concerned about the possible 
effect of this amendment upon the peer^ review and clinical 
tr4Al preliminary .data review systems of the ^Jational 
^ Institutes of Health. Ttie conferees thus wish to state 
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.-^as clearly as possible that personal data, such as individual' 
& medical information, is especially sensitive and should be 
. givfen appropriate protection to prevent clearly unwarr^guited 
invasions of individual* privacy. While the conferees are ^ 
sympAhetic' to the concerns 'expressed by NIH regarding- its 
comnittees' funding recommendations and analysis of preliminary 
datdr the conferees are equally sympathetic- to concerns. exrHressed 
by citizens' groups that important fiscal and health-related 
information^not be unnecessarily withheld from the public. ' 

•^Wi^h these conpetina. interests in mindr the conferees have 
^ secured assurances tnit the appropriate House and Senate 
committees will rteview the unigue probl«ns of NIH tinder the 
new standards. Inde^, it is npted ^that the subcomraittee on 
Reports, Aopounting and Managenent of the Senate Government 
. Operations Committee has already held three days of hearings 
on this matter and plans t;9 continue with' further inquiry at^ 
•an early date." ' 

In its Report of' April 30, 1976, the President's Biomedical 
Research Panel noted the important concern expressed in ^videnc;^ ' 
to the Panel 6vpr the possibility that meetings of peer review 
qf:oups might be openejJ to the general public. The Panel noted 
that: 

"Aside from the obvious likelihood of a decrease in the candor 
of the reviewers and the lack of any indication of how the 
members would be protected professional^ly and financially from 
yengeful.atzrts and charges of libel of bribery, open meetings 
would have at least two other objectionable features. The first 
would be the reluctance of many qualified -scientists to parti--" 
cipate as membefs of Stucfy Sections and Review Committees. The 

^second concern relates to the untenable discrimination between 

investigators from two classes of applicant organizations: 
those- able to send observers because of geographic proximity 
to the review meeting, loaA representation, of financial itieans, 
and those unable to take advantage of the open sessions." 

ttie Panel the refore recannended that .the Public Health Service 
Act be amended '\ . . to prcw'ide statutory assurance that the 
initial review for scientific ancf technicaa merit {"p^r review") 
remain totally confidential." — 

This is in accord with the overv^ielming sentiment of members 
of advisory groups surveyed by the NIH Grants Peer Review Study 
Tern. Only five percent were in' favor of opening IRG sessions 
to the public and *3ust seven percent supported such opening 
simply to the investigator. .Figures were only slightly higher 
for Advisory Board and Council sessions concetnpd with grant 
application review. (Supplement A, p. 56)* ^ 
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C. Privacy Act (PA) . \ " 

* V . r 

Description of the T*aw. ^ ' .^ 

^ This Act, approved on Decerober 31, 1974, and effective in 
pertinent part on September 27 r 1975$ [Pdblic La^ 93-579], 
regulates the collection, maintenance, emd use of information 
J about indiviSuals in agency systems of records. ^ (Appendix F-9) ^ 
The term "Bystem of records" is defined to cover " ^ \ • a ^ * 
group of any records uoJer the control of any agency from whidi 
information is retrieve by the name of the individual ,or by some 
. . . other identifying particular assigned to the individual." \ 
^ The Privacy Act al^p requires agencies t6 establish procedure^ 
under whi6n such individuals ^re given access to ande an oppor- 
tunity to seek* correction or amendment of this information. ' 
♦ • 

' 2. Availability of summary statements and reviewers' opinicms 
^ to the principal investigator. 

On March 8, 1976^ the Associate Director for Extramural Researdi 
and Training, NIH*, fSublished an Instruction and Information -iis I )^ 
Memorandum, 0E3?r 76-1, caicerned primarily with the Privacy Act, 
but dealing as well with the POIA. (Appendix F-10). Itie I&I/ 
^ memorandum indicates that: "All NIH official files for granta ^ 
. • . " constitute a system ofs^r^cords under the Privacy Act. In' 
discussing release of information under the Privacy Act to sudi 
individuals as principal investigators, program direlqtors, and 
fellowship awardeesy the I&I memorandum furtiier provides £hat: 

'"When the subject individual- makes a reqCiest.;Eor information ^ 
^ ffdin a record pertaining to hinv^tr all fact and opinionf * 

pertaining to the subject individual of the record should be 
f released to hiin/ner under the PA. (Biie names Of the outsiidtL/ 

reviewers, i.ie. , jperscMis who are iK>t niembers of standing or ad 
^ f hoc caiinitte€?i fiM v*io, at the request cxf NIH fetaff, ^Submit* 

I , oplhions in writing, are not to be listed on tile sunnar^^ s^te- 

• ^ ^ ment, but are listed elsewhere^in the qfficial record). Jtfi - 

addition, all factual information fi^om the summary statement 
' is available to the subject individual, e.g.r name'of investi- 
gator; title of project, description of project r budget 
' * of project." *^ " ' 

Hie ill menorandum goes on to state that the pr.iority score will 



nob- be released and' the NIH . will release verbatim only 

and FO^. 



v*iat fiS required under both ^ and FOIA. 3/ 



3/ The priority:^icore is not considered by NlH to be "aboOt'* the iijj^'- ^ 
tlgator and is therefore not subject to the Privacy Act, ..Also, 
takes the position that release of the priority score is not "required 
uAder EOIA because it falls within exemption 5. 
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-/With reipect to site visit reports and c^inions of outside 
reviewers persons whose* opinions have been <A)tair]ed but who 
are not members of the review" group) ^ the Z&Z m^norahdunr states: 

"NIH has generally taken *galient information from the sit^*' 
V Wisit reports agd outside opinior^ for ihcorporaton into 
the summary staranent., After informatioh is incorporated 
into the'sumnar^rst^tementr the site Vi^it report and out* 
side opinicxis aje not retained. _ 

"If a Specific reC{uest [by-tKe principal ^nvestigator] is 
made for accgss to these reports >*iile still in our [NIH'sJ 
• possession, appriC^riate- information^, . . from them must - 
. be provided t6 the .requester. . , 

'*T!he name dl^ffi putside reviewer is not deleted from documents 
repres^tirig his/her cpinions^^ e.g. ^ mail reviews, site 
visit notes submitted by individual site visitors. Vftj^n 
Executive Secretaries prepare site vi^it reports and sQmmary 
statements they will not ordinarily- asjsobi ate individual 
» r oplnicMis with reviewers names, however/" * 

Ttie survey of advisory grci^> members conducted by^the Study Team • 
showed that 79 perfcent of those responding favored the Tule set. 
forth in the I&I memorandum pf releasing the summary statement 
to the principal investigator without the priority score.' 
(Supplement A, Page^56.) However,.. a sjobstafitial majority, 60 
perceht,^ believed that the priority SGOPe- Should also be released, 
with^an additional seven percent indicatihg that release 
wcxild have no effect on the quality or efficiency of the pfeer ' 
review system. ^ K hiajoritYr 53 percent, supported fllease of — 
individual- reviewers' opinions if the reviewers rem^ir^d ancyiyroous 
with'c^ added six- percent in the "no ^ffeqt" category,^ but only 
five percent would e^ree to such discl^i^es with "the reviewers ^ 
identified. - " ^ ^ . 

*In fte judgment of the Study TteaitI, these results reflet a 
recoghitsiibn on the part of those surveyed that the^lncipaj.' 
investig^torjhas a right to the complete and detailed record * ^ 
of revietwg^wp' action oh his or her research proposal, f^i^s 
is evOi^nced ^ the respondents '.oVerv^elming support for release 
of theiumj^y/statement; .with .or without the priority score ^ to 
*he. investigator . , On .the other habd, if an individual reviewer 
is identified to' the investiga^ as having .expressed a particular. 
ot>ini6n on the proposal, n^^ther or not adopted* by the review 
group as a v^ole, this is likely to affect the candor of sudi 
, opinions in the^futmre. As a result, those surveyed did not r- 
favor release of sucn opinions unless the reviewers' anc»vymity^ 
was protected.' » . • 



The Study Team believies that, iniasmuch as the review group* 
f '\ nierobers themselves do not believe that sumnary statements should' 
be kept from the principal investigator, there is little justifi- 
*• cation for not so releasing them, whether or not required to do 
so under the* Privacy Act. Also,* for reascms outlined elsewhere 
In this report, the Stpdy Team feels that the priority score should 
not be deleted. • 

With resEject to individual reviewiers' opinions, th? Study Team 
^ recognizes that, in accordance with the Privacy Act, t^ese roust 
nprmally be released to the principal- investigator. In view of 
^ ^ ^the surl^ey^ results indicating that more than half of those 

responding were in favor of such disclosure, provided that the 
rev ievfers*" identity iS not revealed, the Study Team sees no 
substantial reason for objecting thereto. Al: the same. time, 
^ ' the Study Team urges that stqps be taken to establish an adequate 

, * legal basis for protecting the reviewer 'a ancMiymity either through 
reinterpretation ^^existing law, if i^eascmably possible, or 
enactment new peg isXat ion, under cirqumstances in which release 
' of reviewers' opinions is required under the Privacy Act. 

RBOOMMENDATIQNS ;- . ' ~ - 

THAir HBE SUmARY STATEMETpS BE MADE AVAILABLE UPONllBST TO THE 



^I^BST 

0^ 




PRINCIPAL INVESTlGATOBSV INCLUDING THE PRIORITY SCOl 

IHAT IN RELEASING REVIEWERS' OPnjIONS UNDER THE PRIVACY ACT, AN 
ADECjtJATE LEGAL fiASiS BE ESTABLISHED FOR PRCXEECTING IHE REVIEWERS' 
AtJONYMITY EITHER THROUGH RElWTERPRETATtON OF EXISTING LAW, iF REASON-^, 
ABLY POSSIBLE, OR^ ENACTMENT OF NEW LEGISLATION^ , 

Effect of Sunshine Laws on Scientific Progress ' ^ 

It^s clearly not possible at the present time to make an asse 
^ the effect 9f the recent Sunshine^ Laws on the progress Qf fail 
medical research in this country. Ho^er, it Iseems posslible tt 
if invest igajtors conclude that^it is pontrary to their b^J^interests 
to include details of new experimental and theoretical e()proaches f 
in research. pr(^x»als, the vagueness of such applications will ^ 
^.interpreted as weakness in the scientific undertaking.^ This 
CQUld r^esult in the stifling of innovative research and decreased 
support of new investigators -who have little or no background * 
or history of research activity upcxi which reviewers, may judgp 
merit. If so, the very Life blood of scientific progress, namely 
innovation, and youth may be lost.^ ^ 

Thus, It would ^pear e^ential that ^e impact, of this- ne^legisl^- 
tidii' be evaluated periodically so as to assess these factors. 
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. The Study Team believes *that the concerns expressed by the 
7 ' President's Bioraedicai.#setfrch Pane} are well fpunded. If 

NIH were required to open meetii^Sr-it is likely that reviewers 
% ^ woald have reservati^ons about providing candid^ negative criticisms 
, reseatch -proposals. This would undoubtedly result in ambiguous 
; / ^^nd superficial evaluations, reducing the qCiality of the judgments 

. ,,,^t]y the review, groups and thus the quality, of NIH granting opefa- 
tions, \i*iich allocated about $1.3 billion in, fiscal year 1975. 
• . ' Also, some individuals might even be unwilling to s^rve on 
review^ groups for fear of harassntent. ^ * 

RECGMMENEATIO We. ^ , ; ' \ 

IHAT ^TBOSE POfCrforsB OF THE MEETINGS OF ADVISORY GHDOPS WHICH INVCOiVEl, 
THE REVIgW OF GRANT APPLICATIONS CONTINUE TO BE CLOSED TO TOE PUBLIC 
(INCLUDTNG THOSE^SUBHITTING APPLICATIONS), EIIHER CNMl^. CURRENT 
EXEMPTICNS-TO TRE OPEN+-MEETING REQUIREMENT t)R THROUGH. A STATUTORY 
AMENDMENT . A . ^ , ^ ' ' ^ 

/ •'4. Review Gr.oup Membership ' , - * 

Another change in the FftCA pressed in Senate Bill No. S. 2947 
(Referred to above), would be to require that one-third of the , 
members of each Federal advisory group "... be drawn from 
^ j^citizens in private life v*io sh^l represent the interests of 
i ^^the Public ^ . . (AppendixAjT) . if such^a provision is inter- 
^ prated to require the appointment of individuals without sci«i- 

^ ' / tif ic^ expertise to IlCs, this wpuld destroy the fund^gn^l^^^^^ , 
principle of jthe ^lual peer review system, nameiy^thaT'Hiequality 
of research proposals is best assessed by e3?perts in particular 
fielder v*io are scientific "^or technical "peers" of the applicants. 
This Is accCraplished at the first stage of review by the^IRGs. 
Bfoader public input as then obtained at the second level of* 
the dbal' review syst^vrfien the prc^xDsals are considered by the 
National Advisojcy Councils, and Boards made up of leaders, not 
ofily in^scient^fic fields, but als6 in education and public affairs, 



a 



R y^MMENDA'nON * / ' . 

•mT, NOTwf'mSTANDING PROPOSALS WHICH WOULD* RBtUIRE PUBLIC REPRESEN- 
- • cTATICN ON' ALL COfWimES, THE CURRENp SYSTEM OF DUAL REVIEW BE 
. •' PRESERVED, WITH GRANT APPLICATIONS BEING REVIEWED FIRST BY INITIAL • 
REVIEW GROCFPS -CONSISTING SOL^OF SCIENTIFIC. AND T^CHNICAL^EXPEaiiS--^ 
, AND THEN BY NATIONAL ADVISORY COMCILS AND BOAtoS WHICti' INq/JDE 



REPRESENTATIVES OF THE PUBLIC. ». 

V 




RE^DMMENDftTION i . 

THKT THE NIH AND H£W' ESTABLISH A MECHANISM FOR SPECIAL, PERIODIC 
. ASSESSMEHt OF THE" IMPACT OF THIS NEW LBGISIATION ON THE QOALITY OF 
GRANT APPLICATIONS. AND ON THE QUALITY OF PEER REVIEW OF SUCH 
APPLICATIONS AND THAT SUCH ASSESSMENT BE REPORTED TO TOE- LEGISLATIVE 
AND EXECUTIVE BRANCHES OF THE GOVERI^WENT. 
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„• V lfgACT OF REVIEW W0Rtgi3ftD ON CQALITY OF 

INITIAL tERIT REyiEtT " 

a . ■ " _ , 

Over the pa^t several years the workloads* of many of the Initial Review 
«6roup6 (Study Sections) have ihcr'eased to levels which can only be defined . 
as excessive. During a period in which there wer6 seductions in the ' • 
federal workforce, including the NIH peer review workforce, and new 
cohttols werW placed, on the formation and use of advisory groups, the peer 
review workl^idkincr eased in size, and ^xirplexity. Hie denands being 
placed upon already overburdened peer xeview ^roup roenbers and NIH execu- 
tive secretaries will, unless corrective actions are taken very soon, 
result in a lower quality -'of peer assrasmeftts and EecoranenfJat ions. 

I. BftbKGBOUND ' . 

Mentoers of NIH grant review groups have expressed a decided reluctance 
to assume further peer review responsibilities: 97% of study section 
roenbers, 86% of roenbers of other NIH initial review groups, and 90% 
of responding roenbers of Advisory Councils and Boards rated tl?e 
suggestion that initial review group raeitbers' workload be increased 
as poor or very poor, in tHe survey conducted by the Study_Team 
(Supplement A) . In «3dition, there are currently problems in regard 
to (1) the adequacy or summary statements, (2) loss of continuity and 
quality in conmunications within the NIH on peer review issues, and 
(3) inadequate flow of information between NIH si^t and applicant- - 
investigators. The administration of the NIH peer review process, 
including quality control, is becoming more difficult, as the workload - 
increases quantitatively and also becomes more conplex. 

• TOE STODY TEAM IS GONViNCED TOAT WORKLOAD PROBLEMS MUST BE HESOIA^ . 

NOW, TO PREVOtt DETERIORATION OF THE SYSTEM. IN PRESENTING RBCOmEN- 
- DATIONS WHICH ADDRESS TOIS PROBLEM, TOE^TODY TEAM EMftiASIZES THE 

IMEORTANCE .OF MAIMEAINING TOE lOTBGWlY OF TOE PEER REVIEW SYSTQl 

AND ITS ABILITY ?0 FUNCTION EFFECTIVELY.. g.' 

The international respect accorded the NIH and the major role of the 
NIH in biomedical research over the years, have been achieved, in 

irge part, through the accomplishments of NIH grantees, whose research 
was supported by about 60% of the. total NIH budget in fiscal. year 
1975. Since the NIH peer review system is the^rtoary quality control ^ 
mechanism for the allocation of grant funds, the quality and consis- 
tency of the grant peer review. process must receive full respeft and 
protection from any forces Which threaten its- integrity. " 

The Study Team is also sensitive to the fact the "peers" roust be 
considered a resource to be protected. The NIH should not abuse or 
ali«»te this resource, whidi has performed so well over the years. 

tt ■ 
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n. CURRENT STATOS OF TOE NIH GRANTS PEER REVIEW SYSTE M, IN TERMS C» IHS 
WDRKLOM) ' . :— ^ ' 

The wwkload problem ik the first issue to be raised vAienever NIH staff 
. menbers discuss the peer review system. MG has sent f<M:w«u:d to C3D, 
NIH, sev|^ral r^rts of studies by executive secretaries, suggesting 
alternate ways to reduce Study Section workloads. Participants $n 
workshops sponsored by the NIH Executive Secretaries Review Activities 
Coroittee (ESRAC), reported in recent reccroendations to the Deputy 
Director, JIIH,(1) tailing attentiCw to the seriouisn^ss of this 
problem. 

* * » 

The study Team is concerned that review overload either already has 
or will have adverse effects on: 

o The quality of pepr review reconmendations. w 



0 The ability of NIH to recruit and retain outstanding ,experts 
to serve on peer review groups. 

* 

o The willingness of peer group members to take on more work. 

o The openness of peer review groups to unique or "unorthodox* 
research ideas. , 

Q The ability of NIH to recruit and retain individuals of hi^ y 
conpetertce to serve as peer review group executive secretaries^ 

^ o The quality of docuroen^tion of peer review recoramaidations 
and technical critiques in site* visit reports and suionary 
statements^ 

0 The ^ility of Nip staff responsible for peer review to perforin* 
to their best capability. • 

^ o T*ve quality and- extent of the assistance that executive 
secretaries can provide applicants prior to review by * , 
identifying additional information needed. 

. . o Jtie quality and depth of, information provided to grant 

applicants on the substance of the peer review recommendations 
and specific critiques. 

o Ttie. timely review of grant applications.- 

Th€ incrlased wotkload of grant review groups is most readily demon- 
strated in terms of the number of applications per review qroap per 
review session^ shown ih Figures vi, 2, and 3. This increase has been 
accoa()anied by a decline in the review workforce. The number of grant 
applicatiois reviewed by individual I»G Study Sections in l^ay and 
June 1976 for the, September /October 1976 Advisory Council meetings 
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ranged frcxn 30 to 128. lhe .50 Study Sections whicii met at that tiine 
averaged 44 applications per meeting, IW^qty-four reviewed oyer - 
■^90- flppliqaticMis eadi, excluding the revie*^ on an CTierg^icy basis, 
of large nunfcers of fellowship applicaticxis either at that meeting or , 
at a special adjJitional meeting schedule a few weeks earlier, 

Stody Section workloads of 90 or note apt)licaticMis require that the 
meeting last up to four days, often going tn^o evening sessions. 
Reviewers are. exhausted by the end of suchTaKajtbori meetings, Ihe 
ability of even the n*6t highly skilled executive s^etaries to 
capture the substance of the review group discusson of over 90 
applications at it sessicm, acturat^ly^andrin detail, roust be questioned, 

E^^cuti^^&:^ecretaries wh9 are best qualified tO'^bdge the effectiveness 
of rwiew group- performance recoifneid that the number of applications 
be adjusted to limit review* sessions to no more than two and <»ie-half 
days eadi. The Study Team regards this as an appropriate guideline 
in establishing workload ceiling's for review groups. ^ 

Furthermore, the.unr^nerated effort of QfKdi review group member, -in 
preparation for the ^review meetings, averages some three work weeks, 
or 120 hours, of detailed study and preparation of reports, Itie 
quality of a reviewer's preparation for meetings ytith heavy loads is 
visibly less complete than for meetings at which the f^eview load per 
individual is appropriate to the effort (done on a' reviewer's "own-, 
time") required for careful, study before me meeting. 



It should be esiphasized that the number of applications per review 
group per review session is only a partial iodicator of workload and 
far from the definitive one. 5he review process also has become more 
complex arxa the following factors must be considered: ^ 

o As the increase in the NIH budget has taperjed off, and 
competition has increased for limited grant funds, the 
nun4)er of reapplicatioos also has increased/ 

( ■ X . \ . 

o Research has grown more complex: the NIH receives more multi- 
project applications involving complex technology ^Aiich are 
'more difficult and tiroe-peRSuming to review. 



o Related to the above," research budget 'proposa^^ jrtuei^ receive 
' mcjre scrutiny. 

o Increased pressure from applicants for detailed reports of the 
review increases the time required to review each applicaton. 

o Review procedures are more rigorous now and docuro&itation is 
required in regard to the u^€f and protection of human subjectr, 
the appropriate use and care of res^ardi animals, the avoidance 
or containment of hazards to human life and health, and the 
, aivironment associated with biomedical researdi. 




o The requirements of the "sunshlne_laws," discussed, in aoother* 
section of this report, have added to the conplexity. ' - 

TLl. REOOtflBCATIONS AND DISCUSSION , ^ 

1. IHAT THEtblREJCTC«, IHH, TAKE IMMEDIATE .STEPS TO LIMIT THE WORKLOAD 
OF ALL INITIAL (»AWr REVIEW GHOOPS TO A LEVEL WHIC» IS COMPATIBLE 
WIIH MAIMD^INCA Tlffi: ^ QUALIiy OF PEER REVIEW. - ' 

The excessive review workloads inposed on a number of Study 
X « ' Sections and sonvel special revi^ groups in the. awarding Institutes 

cmd Divisions or NIH nwst adjusted downward. Figure 1 illustrates 
the w(xkload trend for ORG. Ihe remaining reconinendations address 
the iH5>lernentation of Reconmendation 1. 

2. -THAT AUnOaTY TO ESTABLISH OR DISOONTINUB INITIAL GRANT REVIEW 
(atWPS, AS .THE PEER REVIEW WORKLOAD DICTATES, SHOULD BE DELEGATED ^ 
TO THE DIRECTOR, NIH. ' • ^ ' 

Congress has given the Director of the National Cancer Institute 
^ authority to establish advisory groups{2). This authority should 
also be ptov^ed to the Director, NIH,^ to Enable the NIH as a whole 
to respond roost effectively to increasing peer review loads and changes 
in the' review workload within scintific and medical fields.^; 

Although trends ^o become apparent over time, it is not possible from 

grant application receipt date to the next to predict with any 
ipcuracy which program areas -will stimulate* large numbers of applica- 
'jOns. The BIDs announce program opportunities through Requests ,for 
lications (RFAs) at irregular intervals and scientific events also , 
late new research efforts in previously dormant disciplines or 
fields. A small number of special applications jit eacif review round 
cannot be assigned to a standing review* group, either because of the 
unusual nature of the proposed project or activity, or because the 
application is subnitted by a review group, member and jnust be specially 
reviewed to avoid a conflict of interest situation. For these 
applications; ad hoc review groups must often be used. If the 
Director^ NIH,,coiuI3 establish and dissolve r^iew groups, as 
recommended, use of review groups for the- short-term would be more 
feasible. 

3. THAT ADblTIONAL RESOURCES PROVIDED PCK PEER REVIEW OF GRANT 
, -APPLICATIONS ^HERE ACXIEPTABLE ALTERNATIVE APPROACHES TO REDUCTICW 

OF WORKLOAD WILL NOT PERMANENTLY AMD EFFECTTVEtY RESOLVE LONG-^ 
STANDING REVIEW OVERLOADS. 

Where the pool of potential review groi^ menbers is adequate, exist 
review groups which are chr<»iically o<rerloaded should be "split" to 
form two yyt more groups as needed. Ihis should be done at once . 




/ 
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TSm study Tfeam recognized that the orgauiizaticHi reqiri^red for peer 
review prcx:eaures cannot be expanded indefinitely. However, np 
alternatives could be identified whidir at present, could be sub- 
stituted' for the additiCTial staff support needed to. maintain the 
qiiali^ of review. , ' ' 

THAT THE DIRECTOR, NIH, ESTABLIsITa PERMANENT MECHANISM TO 
DETESRMINE AND ENSURE AN APPROPRIATE CEILING OR MAXIMUM W0RKI/3BBs^ 
FOR EACH NIH INITIAL GRANT REVIEW GROUP. 

As the upward traid in average number of supplications per DRG Study 
Secticxi illustrates (Figure 1), the NIH has traditicMially expected 
review groqp^to review- v*iatever pcoposals are received by a given 
receipt date. In the past, allowances have not always iyeen made for 
possible limits inposad.by other responsibilities of revi<sw 
grocp menbers, orjtbe effects of fatigue. The Study TJe^ recoranends 
that the importance of these factors be recognized explicitly, and 
that appropriate ceilings oh application loads be set based on NIH 
teview staff experience, in a centrally managed periodic review. 
Once esta&lished, NIH policy should require that these ceilings 
be honored. ^ • « 

The Study Jeam has drawn upon the recotmendations of the fflH Executive 
Secretaries Review Activities Committee (ESRAC) and^ the suggestions of 
ESRAC workshop participants in fonnulating-this prc^al(l^. Bfecaifee • 
each type of application requires a different- level of effort, and 
because a^lications in different (disciplines may be more or less 
time-consuming and difficult to review, ceilings can only be determined 

individually for each grdup. ' . • . : 

_ '~ ' — . * 

THAT THE NIH CONSIDER tJHETOER THERE "ARE PEER FEVIEif SERVICES TO 
tSHrtt^FE, ADVISORY COUNCILS, OR APPLICAWTS WHICH MIGHT BE 
CWOAKiED IF ALL OTHER WORKLOAD REDUCTION ALTERNATIVES SHOULD FAIL. 

The NIH has continued to expand the servic^a-deciyed fcom or related 
> to the peer review process for grant applicaticwis, in recent years, 
in the face of diminished resources and incfeasing review workloads. 
These ^»er vices include providing the detailed tedinical infonnation . 
of review group critiques* to investigators; reviewing, at the next 
■scheduled review Session, all applications receivei^ by or soon Rafter 
a gj.ven xecefpt date cegardless of the workload imposed; accepting an 
unlimited number of reapplications, for a givei researdi proposal — 
if the original or subsequent applications did not receive an award; 
and assisting applicants to assure ttrfl review groups receive the 
roost caiplete application for„ review. . . 

Until about 1968, \*ien research grant funds each year were adequate 
to make awards for nest of the applications recorroended for approval , 
by the IRGs and Advisocy Councils, very few grant apiJlicants requested 
detailed information about the peer review critique of their a^lica- 
tions. Ihose whose applications had been disapproved had other avenues 
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of support or alternative careers readily available to them. Since 
that time, as NIH funds have ceased to grow r^qpidly (Figure 2) and 
the job market for researdiers and academicians has become highly 
^oif^titive, c^licants have a much higher stake in the peer review 
assessment and furkJing o^^tfieir researdi proposals. Alternative 
' ' Wiroes of support are scarce. Detailed information from the review 
critique can possibly enable the"" applicant to restructure the project 
and compete successfully. There are more reaqpplicatims, and most 
unfunded principal investigators demand detail^ information about 
the review. Reviewers and executive secretaries must work harder 
to provide this detailed informaticxi. While the Study Team agrees 
that curtailing such services or providing less useful service is 
undesirable/ this recdmendaticm is included to erpbasize that the 
preservation of the quality and reliability o5 reviews and reconin»i- 
dations per se mu^t take precedence. 
/ 

IV. POSSIBLE ALTB^TIVES FOR REDOCnC WORKLQftP 

Recommendation 5 urges that the NIH consider seriously some alternatives 
for controlling the^ review load of individual review grcx4>s and 
reviewers, but only un^r circumstances v*iere needed r^^rces cannot 
^ be providied to reduce workload. Hie Study Team has review^ possible 
alternatives for" reducing. peer review workload ^ich are other than 
* -trivial. These are: ' 

A. The establishrooit of additional review groi^ in order to ^ 
"split" or share the workload of groups v*iich continuously 

♦ receive inore applications ea&\ round than they can review- 

• effectively. This is the substance of Recommendation 3, 

and we urge that it be done at once, v*iere there are ^ 
sufficient numbers of qualified experts to constitute the 
needed n^ groups. 

B. Provision for junior executive secretaries, preferably new ^ 
health- scientist admin^trators, to work under the direction 
of the executive secretary for the group. Hiis arrangement 
can be a useful training device for new executive secretaries 
and may serve, in some bases, to reduce the duratiw of review 
sessions and/each reviewer^* s review load. The Study Team 
recommends that this e?^>roach be explored.. 

J- 

C. Provision for editorial services in DRG^ or within each BID 
\whidi reviews grant applications in order to assist the 

executive Siecretariasu Such editorial support would help to^ 
assure the qualitv of anrroary stateroaits and conformity to- 
standard formats md other NIH requirements. The Study Team 
strongly reconinends that sudh a system be considered . 

D. Short-temr use of available IRGs within or across BID boundaries 
vAien such t&fieya groups, established for specialized programs, 
could eas^^j^absorb additional workload. 
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B. Creation of Specied Initial Review Groups (SIRGs). This is ' / 

discussed in another section of this rqport. 

P. Contractor-conducted peer review . .There are precedents for 
contracting out peer review, "ihe Office of Life Sciences, 
NASA, for exarnple, has contracted in the past with the i . 

American Institute of Biological Sciences for sc^ffentific and 
tedtmical merit review of research proposals. Properly 
staffed and i oriented contractor organizations cart provide 
peer review of high quality.' Contractor staff are substituted 
for agency staff, and do not count against the agency position 
' ceiling. The doifcr 6ost of contracting out more than a small 
part of the iJlH graRt peer review operation would be signif i- , 
cant, however, and close NIH staff overview and coordination - 
would be required; ^sparing of Staff positions would be 
partial at best. . - " • 

The l«H has every reason -to preserve and upgrade the present - 
system rather than to replace it. From the WOoldridge Comnittee 
to the Rand Corpocation, mos,t groc5)S which have studied peer 
review have reaffirmed the value and quality of the system. 
It has served as a model for 'other agencies and programs to 
emulate.- The Study Team recomnends that the contracting 
alternative be reserved for special uses with appropriate NIH ' 
overview to assure conformity to NIH peer review policies and 
procedures. ■ > ^ , ' 

G. Modification of peer review procedures to permit review groi^ 
* and staff to absorb greater workloads. One such change has 
tecently been inplemented ; with the change to the new fiscal 
-year, the review cycle has been lengthened fpom six months on 
the average to eight montiwr on the average. This action, ^ 
while predictably not popular w*ith applicants, does give 
executive secretaries more time.in vAiic* to schedule site 
visits and prepare sumnary stSiteinents. Other possible 
modifications are: " • 

1. To decrease the time and effort required to review each 
application by staff screenj(hct»and rejection of inconplete 
or incoherent applications; to curtail discussion of 
applications judged by the primary reviewers to be 
either excellent or without merit; to curtail. or eliminate 
site visits; etc. Casual, observers of review meetings • 
often make sudi suggestions, saying, for example, that 
too-much time is spent^jy the primary reviewers in 
reading or paraphrasing the proposed project which alL ^ 
have in front of them. 

The responses from executive secretaries, and from grant 
peer review group members (see Table 16, J>age 61 of 
' Survey Report, Supplement A), suggest that full^eview^ 
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of eadi application is generadly held to be more 
.-inportant than a rapid rate of review which might be 
. at the expense of the quality of the review. Provided^ 
a teview* groyp is working effectively, the opportunities 
to spare peer review resources through sucii review shdrt- 
cuts are yery limited, and b^t not pursued. Further 
staff Initiatives to cull inadequate applications might 
be explored. * - . ' nJ . 

TO oirtail staff-screening of applications by executive 
secretaries to assure that ^licatic»is are conplete and 
clear before they go to the review groups. Hiis service 
to the applicant researcher is a useful function of the 
executive secretary, intended to assure that the applica- 
tions are fairly reviewed rather tii^ to save time or 
effOTt. It is a valuable service whidi NIH review staff 
renders to grant c^lication principal investigators, as 
wo^loads permit.^ We do not reconinend, except^ in case 
or dire necessity, that this staff assistance to the 
^lic^mt be curtailed ot abandcxied. ' ' . 

To queue 'applicati<xis. Ihis practice has been proposed 
by DRS in the past. All a^licat ions, beyond the detet- 
mined review c^^city per round of the assigned review 
group would beJield to the next review cycle. The NIH 
has rejected this proposal in the past, adhering to* a 
long' traditicxi of providing prcropt review for all 
applicants v*iose e^^plications reach 'the NIH on or before 
a given recei]^Bftte. , Nevertheless, it is felt that 
queueing is pi^Brable to acceptance qf poor quality of 
reviqipr since it is essential that ,ortly meritorious 
projects are supported. Prom that perspective, the- NIH 
cah afford to reject queuing or any other "sudi heroic, 
dis€asteful measures (xHy if alternative ways of reducing 
or adjusting review workloads can be invoked. ^ 

To reduce the number of replications, after an appli- 
caticm is reviewed but not awarded. The review workload, 
can be contained scmev^at if the number of such reapplica 
'tions vAiidh will be accepted by the NIH in b^alf of a 
given individual can be limited, perh^ to one per 
project. Since lall investigators should have 6very 
opportunifcy to1:eapply in order to obtain support and^a 
revised application is, in essence, a 'new applicatioh^ 
this alternative is npt acc^)table to the Study. Team. i 

A more accQ>table alternative, where an unchanged appli- 
cation is resubmitted with the hope that .it will compete 
successfully for funds the s^econd or third time around. 



' .136 _ l.1<j 



T 

is to forward the applicat^ion for BID c»nsid«Mtion, 
013 the basis of the application's prior merit review. 
Applicants usually do benefit from the evAiation- of 
the first application, however, and second or»sub-^ 
sequent submissions .are most often revised to take 
advantage of 'information frati the earlier review. 



V.' OTHER WORKLO AD-REIATED FACTORS 

• — — ' ' ■ ' ' 

There are factors which^ate crucial to the quality and consistencfy^f 4 
mi grant application peer review to which the NIH has not p*3ressed 
adequatejeffort ih the past. To iirprove the efforts^ tn^ese ateas 
would ?^ire assurance that the review workload ^se<E executive 
secretari^ is not so large as to preclude the involvement of these 
key individuals in quality control activities. These activites 
include: ■ <- ^ 

o Formal orientation and continuing education programs, and an 
' -adequate apprenticeship period ^or executive secretaries before 
* ' ' assuming full responsibility for a review group; " • 

o An effort to assure that the peer review process is reasonably " 
consistent across the NIH, for such matters ^ the interpreta- ^ 
tion and use of the priority rating procedure, the concepts of 
" "approval" and "disapproval," and the full participation of 

review group manbers in discission. , * * - ^ 

- - Tt^B Division of Research Grants has attempted to fcompensate ffit / 
variability in the rating behayior of Study Sections,, over tilne, , 
' by computing "nobialized" scores. This r«pfes^nts ah effective . 

but indirect approach to the problem which has certian limitations 
- One direct approach would be to provide opportunities for 

executive secretaries to work together, to observe and learii ■ 
from the review process of groups other than their own, to • 
obtain appropriate training, and to have available information, 
for tliiTiT'own use and for their reviewers, vAiich- would provide 
basic .definitions, review objectives, procedures, and criteola; , 
and which- would be reviewed with each new reviewing or staff 
manber. p 

' o Comnon standards and pr<5cedt5fes across the ntH for project site 
visits, criteria for detenninbg when a site visit is needed, 
and the content antL<fofflSat^ site visit reports. 

_Many of these suggestions.^«^isca*|iaii#ls6where in thife report. 

' Vt. THE "SUNSHINE LAWS" ^f^ . 

Ihe requirement for more detailed-'jdoGamentatipn of review reconmenda- 
tions and more careful management and retention of review iaforraation, 
^nerated by the Freedom of Information Act and Privacy Act, has 
• increased the complexity of the peer review workload and has* added 
somewhat to the effort required to process each application. These 
laws are discussed in anothel^ section of this report. ^ 
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ON QPP^Tl OF INITIAL MERIT REVIEW 
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1*^ Kaufman, A'.'a^J Mliaffino, S. Merooirandum: ESRAC Repannehaations from 
; * / E3ttriiwral-CollaBorative Rfet^^eat ]am ESRAC Summary Statement' Workshop. 
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NUMBER 
*5,0Q0 



TOTAL NUMBER OF APPLICATjqNS J(R-OI A(4D' RCPA) REVIEW£0 BY ALL 
ORG STUDY SECTIONS (NfiS^CH.I969^0CT0B^ 1376) . * , 
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NIH-Wn^ STftNDARDS AND OJIDELINES 
FOR PEER REVIEW PROCEDURES 

_In order to provide a legal mandate for the NIH systena of sciehtif ic peer 
review of grant applications, Congress amended the Public ^ealth Service 
Act in 1974 by adding a new Section 475. This section directs the 
Secretarylof DHEW to require, by regulaticm, appropriate scientific 
peer review of applicatiwis for biomedical and b^avioral research » 
grants under- the said Act. Section 475 states that the reguiatiais sljall, 
to the ext«it .practicable, retire peer review to bp conducted in a inanner 
cxmsistCTt with the NIH system in 'use at the time Section 475 was enacted, ^ 
»d by peer Review groups performing such review jDn or before that date.. 

In accordance with the aforegoing statutory mandate, DHEW has, pubj)Lshed a 
notice of propoeesd rulemaking in the. Federal Register (Appendix F-2:) v*iich 
sets forth proposed regulations r^arding the use d£ and criteria for NIH 
grant application peer review. 

Although several of the BIDs have developed guidelines for their own use, ' 
NIH-wide guidelines and st^ards governing specific oper3ting procedures 
in peer review are incomplete. It seemed appropriate to determine v^ether 
general guidelines, applicable NIH-wide, are needed, and if so," to develop ' 
the appropriate documents. Accordingly, the Associate' Director for Extra- 
nural Research and Training and the Grants Peer Review Study Team asked 
the Executive Secretatires Revlw Activities Committee (ESRAC) to conduct 
sue* a study by collecting and i^tifying all current written policies 
and procediites developed by^IH ar^the various Bps .and to present ' . 
recannendations regarding their applicability for general use by ^1 BIDsv ^ 
The £SRAC report is due early in 1977. It is expected that it wiir provide 
I?IH with the documentation n^^sary to determine the need for additional 

^ NIH-=vi0e guidelines oc standards beyoriTlJiose. already con^iped in the 

'proposed regulations. 

"Ulflt Study Tectfn believes that an inportant Outcome of the ESRAC study .w^^ll 
be the identification, for all BIDs, of the staiadards and guidelines 
utilized by particular individual BIDs, some of v^ich may be appropriate 
for utilizaticxi NIH-wide. . It is recognized that .there may be instances 
whidi 'require a particular BID to have certain policies based on its own 
particular needs.^ Nonetheless, it is felt that by providing BIDs with 
information as to the manner in v*iich each of then functions, all will , 
benefit from the knowledge gained by their coliect^ive experience. 

Furthermore, it is a sine qua norTSIat the quality of peei; review and otx . 

• administrative practices foTTcwel by BIDs in regards to research grants ' 
and other award instruments is directly related to the staff which has the 
various r^pcmsibilities. In order to provide and maintain high quality 
staff activities and to assure its continued functic»iing at the highest 

- level, an on-going quality control program'is essential. Central to this 
is 'the nfeed for formal training both for new staff arid for those v*iose 
duties may becontLjrouting. In add it iai, frequent discussion groups for 
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s^fV during which open and free ccniminication can occur are also 
important training experiencal. 

Ihe quality of peer review is d^ndent not orily on the professional 
NIH staff, but more inpprtantly, on the advisory groups without .whom the 
system could not function. It should be recogniied that education of 
roenibers of Initidl Review Groups (IR3s) and Councils is essential and is 
a continuous process. Although guidelines are available in/the form of 
a handbook for advisory groups, orientation sessions for both new and old 
meoiDers of t?G3 and Councils are important in ordei: to «i(iiasize and 
reesnphasize the responsibilities of each individual in regard tio review 
and the need for grcAjp interaction, * J 

^BCOMMENfiATIONS ' 



1. TOAT TOE PIOP0SED,PEE» REVIpW REGULATIONS BE PINALIZED AS SOON • 
* • AS POSSIBI£. * 

I 2. IHAT NIH STANPARDS AND OJIDELINES BE PREPARED OR REVISED AS 

SOON AS POSSIBLE AFTER COt^IDERATIO* AND EVALUATICW m : ' ' , • 
■ ^ RECXMffiNDATIONS MAEE IN TOE STODY BY ESRAC. 7 

% ' ' ' ' 

3. THAT, IN ORMIR TO MAHflS^N AND IMPROVE THE LEVEL OP EXCELLBNCE 
OP THE GRANTS PEER REVIEW SYSIBM AT NIH, :A PHSONNEL ^ITION " 
, SHOULD BE ESTABLISHED WITOIN IHE OFFICE OP EXTRA»«RAL RESEARCH 
^- , AM) TRAINING TO PROVIDE FCK QUALITY' ASSURANCE OF THE SYSTEM. ' 

SUCH A PERSON SHOULD HAVE STAFF RESPONSIBILITY PDR^'MATIERS 
PERTAINING TO TOE FORMULATION, DEVELCffMQJT, "n»TERPRETATION,' 
AND APPLICATION OF POLICIES RELATING TO IHE ASSIGNMENT 'AND 
OVERALL SCIENTIFIC AND TTCHNICAL REVIEW. OF RESEARCH GRANTS - . 
AND MANPOWER DEVELCXWNT PROGRAMS. 
• i' ■ % . ' 

4.. -THAT TRAINING CURRICULA BE DEVELCS>ED BY NIH FOR EmAMURAL PROGRAM 
' AND REVIEW STAFF IN ORDE^TO PROVIDE GKIEKTATION AND TO REFRESH 
AND REITERATE PRIlJflPLES -CONCERNING THE PfalDOSOPHY, OTJBCTIVES, . 
■ , -AND PROCEDURES FOR PEER REVIEW. ' : 

■ ■ ' ' ' 
5. THAT OPEN FORUMS AND WC»KSi»PS 'FOR PROGRAM AM) REVIEW STAFF BE*' 
/ ' ESTABLISHED ON A QDOTINUING BASIS SO AS TO ENo6uR?tf3E AND. IMPROVE 

\ EXCHANGE AND CO^fKNICATION OF IlffiAS CONCERNING ISSUES RELEVANT 

N TO THE PEER REVIEW SYSIBI. 



6. IHAT'TOE DIRECTOR, -NIH, THE BID DIRECTORS, AND IHEIR STAFFS tlOID 
ORIENTATION SESSICNS ANNUALLY PQR ALL IRG AND COUNCIL MEMBERS, . 
PARTICULARLY NEW MEMBERS', IN OroER*^ PLACE PEER REVIEW fii. 
PERSPECTIVE AND TO INFORM REVIEWERS OF IHEIR FlBiCTIOre, DUTIES',- 
AND RESPONSIBILITIES. ' , . 
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NIH-WIDE STAIDARDS' AND GUIDELINES FOR PipH REVIEW PROCEDURES 
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PEER REVIEW QF PROGRftM PROJECT AND CEWTER GRANTS ^ 

Peer review procedures for program projec.t arjd center grant plications 
have te«ived and ccMitinue to receive intensive examination NIH. staff to 
assure that the same general, principles of review are followed for the 
caapleiC large grant applicatidns as for ,the traditional investigator-initiated 
research grant applicatiwi. 

Ihe^per captions of the bloroed^^^^J^coninu^ regarding program project and 
cmter grants focus^ on a number of issues, sopie of v*iich relate to peer 
review of such gr^nt. applications and others vihich are more perifAierally, 
if at all, related to peer review. Of those who addressed the issue of 
program projects and center grants in letter r^spaises to the Study Team 
or inr testimony at the public hearings, the ifQllowing concerns were most 
frequently roentiwed: ' ' y ♦ 

(a) the poorer quality of peer review of program project and cente'r 
grant applications 'as ccifpcired to that ofTRe traditional scientific 
researdi grant applications; 

: ^ * ' A 

(b) inadequate separatic^ of review and program functions among 

^ BID staff which leads to bias in the review as a result of pressure 
in reviewer selection; * ^ 

^ ' ' ' 

(c-) the. appearance .of conflicts of interest for or against a program . 
project or center grant among site visitors reviewing sugh applications; * 

(d) adequate balaqce of funds betweert traditicMial and^program project 
and center grants aNd'^e increasing allocation of funds for the latter; 

(e) the lengthy duration of committed support for program project and . 
center grants, thus closing off funds available tor initiation of 

new regular gr-ants. ' ' • 

NIH review group members also were concerned about the adequacy of the . 
^reyie*f. of program project and center grant applications. The survey 
questionnaire revealed that. Whereas 57% of the respondents thought that the 
review of traditional research grants was excellent or good, a significantly 
lower perc^tage {18%) of the r^pondents thought thatsthe review of 
.program project and center grant applications was extrellent or good 
(Supplonent 1) • ^ ^ \ , l %J 

' ^ . . \ J . . ^ 

The proportion of research grant funds awarded for^progran projects and 
Centers varies considerably among the BIDs. Nevertheless, it sihould be i 
pointed out that, in FY 1976,"*all BIDs except the Divisicai ofc Researdi / 
Re;30urces (DRR) and the National Institute of Environmental Health Sciences 
(MtEHS) awarded ijvore than half of their research grant funds for iiidividual - 
^research projects. The National Institute of Allergy arsd Infectious Diseases 
. (NIAID) and the National Eye institute (NEI) awarded 91% of ttieit research grant 
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funds for research projects, followed by the Naticxial Irstitute of Arthritis, 
fletabolism and Digestive'^Disord^JNIAMO) , 87%; Natiaial Institute of 
General Medical Sciences (NIC36), 77%; National Institute of .Child Health 
and Human Development (NICHD),uJ72%; National Institute of Neurological and 
CoTOunicative^ Disorders and Stroke (NINCDS), 68%; and the National Institute' 
of Dental Research (NIDR), 61%<"^e Naticaial Cancer Institute (NCI), 59%; 
National Heart, Lung,rari(i^^^ Ihstitate (NHLBI) 54%; and the National 
Institute of Aging (NIA), 54%, 4w/arded slightly more than cwie-half of 
tiieir researdi grant funds for researdi projects. The NIEHS divided its 
awards alroost*fevenly between research projects and program projects and 
centers. Almost all of researdi grant funds (excluding General Research 
Support Grants) .were awarded for center grants reflecting the particular 
4 mission of that organizaticMi. . ' 

Peer review procedures for program project arid center grant applications 
have been the ^subject of a further recent study by an NHi staff gjroup 
(Append^ix G). Othej^ studies are also in progress to examine a series of 
issues relevant to these coiplex grants. • Itie, concerns expressed by the 
biomedical comiuni^ reinforce, the Study Team's reconmendaticxi that 
the aforenentipned studies be continued. In addition, it is strongly 
recoran«^ed that there he strict adherence tb the principles, of peer review 
in order to assure the highest quality of review for these ccnplpx grants. 
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PEER REVIEW OF PROGRAM PfiDJBCT AND CENTER GRANTS 
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REVI^ OF 'BUSINESS MMO^EMEHT PRACKCj^ ^ 

• * v / ' ' . . • ^ ' ' • / 

. '* In developincf its assessment o£ the peer review prctoess preparing - 
^ its report, 'the Grants' Peer Revifew Study Team (GPRST) felt that a ^ 
; study of review proceduref # partjiculaifly for large, conplex, multi-v^ . 
faceted granjt3>. in Tegajj<^"to th^ business management practices of tfie " 
^4nstitutions applying 2or such gtants was e^enU^^ Accordingly, 
^ a subconinittee on business management practices was establishjed an^ \ % 

•perfontfed dn in-depth study vrtiich is prxJorsed by the GPIST arxJ is 
incorporated in this section. * ' . ^ 

\. . Background * , ^/ * ' 

Uie Subcooitiittee was charged with the i^^onsibility of 'studying 
all iaspects of the business management issues associated wfui th^ 
_peer review process a^id making reccmmendations based on its ' ^ ^» 
findings. ^ 

In considel-ing various approaches to making the assignment, the 
V Subcommitt^, in its first two m^^tinga, made several bssic<3eter- 
' ' minations, sudh as:** f 

AJ^ Since the issue at hand caiqerne^ peer review and not , staff 
^ ^ review, the study would' actually -deal primarily t^^ith the NIH. 

use of outside business managem^t consultants at the levei^ 
. of the initial review g^oup or project site visit team. As 

* intended here, the terfh> business nfisuiagenen^ ccxisi^fcant, relates ^ 
; A to any individual \*a> bec.ause of background— training artd/ 
ex^rience— is ^qualified to proy-ide expe^rt evaluati<m arx* 
'^'0 ^ ' advice^in assessing ^grantee., instifptional business manaq^eroftt 
^ ^ 'Systems/areas such as .fiscal adj^istration, budget planning 

and execution, personnel, prOoj^lnent,' facilities management, 
^ , property, etc. , . 

B.* The. grant applications that da feature or would appear ;ifb ^ 
r€<^ire bus^tiess mahagenent input as a part of; the peer rev.iew 
process \*xild generally be of the program project and C^^ter 
types or those involving consortium Arrangements and, ^cordingly# 
; ^. the study would be essentially limited to these. " ' 

' -6r^ It wa/ recognized that, in the initial reviw of the larger ^ 

^ and more complex grants, there ia often need for input from^ 
other hop-scientific reviewers whcfee bcfC^oOnds expertise . 
fall into thtp area, of senior l%^el prograirrlnwnagement, i.e., . 
. r ■ anemic deans, otc HXirther, even though there c^h occasionally. 

' l^e some overlap of program management and *usiijess management; 
- ' ^ consultant review activity/ depen^Hfrig tTpon individual qualifi- '-^ 
, * cations and spe^f \c circumstances^^no attempt would be made 
. . to include the role>o#*tlieyrogram managSrtent consultants in 
^' ♦ the stucly. . - / * — . ^ 
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D. ^le substantial extramural operational 'background experience 
was present 'among the member^ship of the subcommittee , a gr^at 
dead :oi^ iniormational inputs from out$id^ would be re<^ired ^o 
develop a conplet* picture of the NIH involvonent and/or . 
coRinitinent an this area/ " ^ ^ , ^ * ^ 

Bie major fact-finding activities ^ the grpup%<^rfe pursued in a 
series of inter-^elatsed phases with the various inputs building^ 
\jpos\ one another. The primary efforts were as follows:' 

A. A preliininary search was ^ made foe^J^IH dcfcuments (Manu^ir^ 
Chapters, etc.)' which might have refetertee to the businiss 
irianagement aspects of peer , review and, 'ftan withici the organ- 

• izations represented by the mentoersliip, a collection was madef 
. of any pertinent BIp j^r review procedural directi\tes or ^ 
•"guidelines. ^ " ' - 

' ' ..^^ • ^ ' , . 

Since th4 above pfovided only an extreq^ely limited indication 
#of what wad«^ing done at NIH, it was determined that a 
standardized questionnaire should be developed for soliciting 
information from each of the BIDs. ^e qu'estiDnnaire was 
.designja^ jto generate specific data on the eiftent pf non-NIH 
business mana^hertt consultant utilization (^ee Appendi^f^H-],). 
Along with respc^ises to eleven questions,, the dpciifl^t requested 
submission of copies of any writt^ guidelines' routinely 
• provided t*o. business jnapagdtienb consultants prior to partici- 
pati9n in si*e visits and, also, sampled '6f Pi^ject Site . . 
Visit (PSV), reports and/or InitialrReview Group (IRG) Summary ' ^ 
Statements ref lectins^ specif ic contributions of the business 
consultants. Itie responses to the (questionnaire were^urrmarized 
and studied 4^see Appendix H-2). ,\ . - ^ 

C. . While the data were being collects from the Nlfl Bins', it was ^ 
decided that an attempt should be ma^e to, obtain- input on 
similar activities whicJh might be involveS ,in the j:eview\6f ' 

/ grants by some other agencies with missions ccniparable to . 
NIH. At a meeting arrange with ^taff of the National Science 
Foundation and the Afpohol, Drug Abuse, and Mental Health 
^ . ftiBministwtion, comments of the visitors'were quite interesting 

• but indicated, that littl'S^or ho business minagenent* expertise % 
I- is built tnto the grant review processes of the two agfencie^^ , 

(see. Appendix H-3)'. ^ . ^ * , ^ • 

D. It was felt that there would- b6 pofe^ntial valul 'in obtaining * 
input from a representative nctabier.of individuals' 'have ^ ^' 

^perfortped ^or NIH in a business manag«nent capacity within the* , 
, grant peer review pro<iess! .An alj-day meeting was, ari*ar)ged[* 
'/ with* seven such consultants with NIH experience- ranging frortk 
. v^ry extensive to comparatively' limits <see Appendices H-4 and 
H-5). ' ^ . ' 
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• de session was fell to be ouite beneficial, providing ample ^ 
Qppoj^tunity to learn imlch of the way in v*iich these people 

„ vi6w, aihong other tilings: (1) the role they are called upon 
, to perform; ^(2) the degree of receptivity to their efforts; 
(3)-. the type 9f'^report v*iich" should be written; and (4) the 

• possibility for changes in' approach v*^ich might make their. - 
^ p^ticipation. more effective. ' , 

'E. Having heard ^extensive canmentary by this group of c6n^l-»» 

tants, the -Suboanmittee determined that it would be of interest 
' to*r€view.**w6 or more examples of IPG Summary .Statemaits ar0 . 
Project 4^ite^ is it (PSV> reports to which each of the indtviduals 
had' contrjMuted. With the help of at least six BiDs, sixteen 
sudi ibcuments were collected, duplipated, and reviewed^ 
In an attempt to display possible differences in approach, 
exan?)les were Selected wh^e the individual consultants had 
wOtked for different BID^r participated in reviews ,of different 

; types* of graht applications. 

F. Because of the iraporlant role played by. the NIH Executive 
Secretaries in the total grants review process, it* was 
decided to obtain input from a representative number of sudi 
NJH staff on the 'same issues which had been discussed with the 
business management consultants '(see Appendix H-6). To allow, 
for freedom of expression, arr^emmts were made for four . 
Separate meetings with appropriate staff responisible Tor 

• review from E»G, NICHD, NCI and HiLBI. With .real - 

. idifferencfes of personal ppinion being shared by the Executive 
Secretaries, these meetings were found to be very enlighten irig. 

' As a- f inal .fact-f indiijg effort, the Subcomnittee decid^ to 
. ' approach, another group of .individuals believed to be ^uniquely 
quaiifi^ to rectot to qt^stions c3h the role of the business 
managonent cpnsultan't in pe^ review. Ifeis would be a number^ 
\ of sentor.NIff scientific consultants v*io have either chaired ' 
, ^ permanent or ad hoc initial review committees or project site } 
'visit teams, or both. Tq contact such' individuals with-the 
lea&t incwiveii^e to them, it was determined that a tele- 
j*ione sutvey would be made# utilizing ccxiference calls sp- - 
thgt two or three €ubcompnvittee members copld spealk with each ^^ 
. the chainnen. . 'A standardized list of questions was deC^'elc^)ed 
in- advance to .use as a guide in querying' the* eight chairmen* 
with v^om conference ^call^airAngements -were made. This approach 
wcx'ked quite ^weir and muoi lielpfui information was obtained ' 
V^.( see -Appendix H-7). ^ * 5 ' ' ' 

iteCO MMESQATIO WS AND DISCUSSION ' z ♦ / 

— ; • i — ' " . ; 

No listing of general findings has;, been presentee? previously . ^ 
b€^j2se each of the recojnmendations that follow Will ^have art 
^"^''ascai^Janying stat^neh of rationale. ; These .statements wijl. 
indyde significant evidence of v*iat was found by the Subcqnnittee 
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in its various Erases of information-gathering. The reconin«>dations 
have^ b^n developed because 61 the strcxig conviction that the ' 
acti<;»is prc^xised are cjritically n^^ed* to fortnalize an NIH 
position on^an impqrtaufifr, aspect of the grants peer rev.iew process , 
whidi, in th? ' past/^as* been left largely to \tim or chance, -Bie 
reconiniendati(»is are: -- v ' * * V * 

A, .tHAT IT BE ^ECOOJIZED^AS m ^1^ POLICY POSITION THAT TH^OSE 
/ OF BUSINESS m/lPGBH!^ CC»JSULTANTS ^AS REVIEWERS ISr H NBCES- * . 
^ SftRY ADJUNCT JO TOE* SCIENTIFIC REVIEW^ OP THE, LARGE,* COMPLEX 
' AND MULTIFACeSeD PROGRAM PRpJBCtS, RENTERS/ OR ^THOSE GRANTS 

INVMiVING CONSORTIUM ARRMK?EMENT^. • ' 

Itieevidencexdeveloped^ through 'the questionnaire indicates 
that the use of sucf^^ consultants in the review of program^, 
>-p^o]^tCand center' grants applications ranges acrosq the 
ntult^^da BIDS frofn "ne\^er" to "almost always,'^ These are' 
perhapsjndt equal extremes ^ but it is concluded ttiat NIH should ^ 
prese^more of a i;pasonable "middle, ground" picture on^thls 
issue with, at leagt, no BID being permitted to determine on' ^ , ' 

owD "never" to av^j.1 itself of the ben^it of such, mahag'e- 
^nt expertise. Thai dis<3ussibns with the business management . ^ 
consultants,^ the Executive Se^rretaries; and the, IRG or^'^^SV 
.cb^irmen provided ample support for the basic value of aug- ^ 
melting th^ highly specialized scientific review'with appro-, 
priate evaluation of business manag^nent practices, ^ 

; , r . ^ ^ ' ^ ' ^ ^ 

B, - THAT THE PRII«:iPAL CRITERIA USED TO* DETERMINE THE NEED *POR . ^ 
' BUSJfffiSS MANAGEMENT CONSULTANT SUPPORT BE .RELATED TOt IT® ^ 

ORGANIZAnONMrOR ADMINISTRATIVE COMPLEXlf*^EXISTING IN AN^ . 
' ? PARTICULAR GRANT APPLICATTCN. - ' ' % ^ 

This reconmendation is based on tjie understanding that there iS 
no sinple Way , of determining vrfien it would bft* appropriate to 
utilize bus ijiess -management expertise 'in reviewing an appli- 

• cation. ^ Certainly'r sheet size of the program pr proposed' 
dollar level cannot^be used &s hard and fast criteria.. It is 

^ in keeping with the many inputs received to say th^t eadi 
application, must be reviewed on its own with a judgmental 

determination being made as to corplexity, 

* ^ '- ~ 

'C, TOAT, CONSlDEiaJG.THE NATUR? OF THE TFftDITltWAL INVESTIGATOR- 
• INI*riATED RESEARCH PROJECT APPLICATION AND I^ REVJEW,, IT BE 
RECOQ^IZED THAT THEVUSE OP BUSINESS MANAGEMEOT COt^TANTS" \ . 
WOULD NOT USUALLY BE CONSIDERED NECESSARY OR AP^R^RIATE, 

Such a 'position was uniformly held by all indlvifltfcgVs consulted, 
♦Hoyjjjj^er, it was expteesed With the understanding that the 
individual research grant application could represent a very ' . 
large and ccnplex project, particularly if it involved a . 

* consortium arrangenent, and might be equally in n^ of ejcpert 



D; that life DEVEUDP A POLICY ISSUANCE CONCHWING THE ROLE OF AND 
* ^NEED FDR ASSESSMENT OF BUSINESS -WyHAGEMENT IN THE MVIfiW OP 
SHE LARGE PRX5PAM PKXJBCT AND CEajTER GRANT' APPLICATIONS • \ 

Along with the three previous recgwroendat ions, this is a • . 
natural follow-up since there is currently no NIH policy \it\i^ 
speaks to the issue of business managemeDt review'. It is felx' 
that roost of those consulted endorse such a proposal, ^s long 
as there is no attenpt to narrowly define the circumstanoes ^ 
under whidi roanagonent consultants are'to be used in the peer 
review process, , / ^ ' 

,E. THAT NIH DEVEDOP UNIFORM GUIDELINES TO BE FOUJ0MED %Y THE 

BUSINESS MANAGEMENT CCmJLXANTS IN IHEIR BDLES_ AS MEM^PRS OF 
PROJECT SITE, VISIT TEAMS AND ADVISORY GROUPS, * * " 

In Spjffce'^of the extensive use of business joanagem^at consul--^ 
tants by certain components of NIH /there "have never been any 
Nlltwide guidelines developed to indicate what is expected of . . 
th«!. Hw^e with whom the Subc6ninittep mit ind icated- J;hat they 
had deyeloped their own approacAies to dqing the ibb-^aSd were 
scinetiines left, wojdering whether they were turnffig in a ^ \ 
sAisfactCry piece ofji^orkr 1t»e Executive Secretaries find PSV 
chairm^ cofifirmai that they provided little verbal guidance, 
if. any, to hhe business management 'consul tants. .As itligHt be 
expected^ the review of multiple summary stat«»Qits and. PSV ^ 
reports indicated vast differences in ^roadi, ^ 'There was a 
uniformly positive response from all quarters td tl>e idea of 
NIH guidelines .being developed, ^ v 

\ 1, That the coriStiltant's'functioh should be to review arid 

• , recommer»d primarily on 4iie adequacy of business managenient 
^ ^ ^ practices and arrangemaits (incruding siKindness of Jxidgetary 
planning) for the prog^r am project of the centex: itself. 

2. That the business iffSiig^R^^ should not 
beton(^aied with e^tolQring'the histdry, oirtenf financial 

f status, or attaini'strative adequacy of the institut^ion as*a 
v#hole unl^s special si tuaViOTS call for it. 

3. Thai, in following the standard site visit approach of 
separating away^'from the scientific xw^, €h'e bi^'in'esd^ 

• \ managem^t consultant should concentrate* on doalirig with 
institutional/administrative staff memberS*v*i©'will' be 
' ' ' ; * directly involved with the project . ! ,^ 



4. * llfiat flexibility shquW be permitted .§0 that, d^)€«cling 

upon tlje lirtdividual ihd ^"tehe specific ciroimstance ot his^^r 
" participation* the busir^sst^njanagemen^-conkiltant coul<f't)e 
' asketfto^ contribute totother administratively oriented 
. aspects of grant ^licatioa review^ such as propo^ 



; prcxiedures for decision-nakingOarrl r.esouirce' allocation 
^ - within a prograrn pft)ject or plans .for controlling access 
* • to and use of shared facilities' and services in a core 
center or resource grant. ' ' 

-^1/2,. 3, /ffri*4' have been tieia together because of 'Similarity 
and relationship to the basic reconmendation concerning 
developroent of performance guidelines for ''the business manage^ 
ment corasultant. These four recotmendations present^new 
ccjKrepts whicii are based on general concurr«ipe from the group 
of edraini^i;ative consultants. Regardless of»tJje historical 
.background for it^ some qf -the BIDs,have been responsible for 
fostering air institutional review approach on the part of the 
roanageroertt consultants which frequently expends mudi of their 
effort in area§ of .little profit. It is apparent tha^ the^ 
great bulk, of the large grants are awarded to the larger and . 
well established iijistitutiong,. ,«^rther, in^ lool^ing- at the 
••large grants^ pictiSfef it is understandable that many of , these 
institutions might 'have as many as 10,, !20,>of more conpetitive*- 
reviews pn the large grant applications ^each, year^. > Therefore, 
as a gerteral riile'^ it may be safe to say that .there i? little 

• real value in having the dbnsultants , devote significant time 
to interviewing »top institutional officials r i?^. /^presidents r 

^ vice-pr€?idents for adm'intstratipn or research cqn5rt;rollers, 
etc. ,^ and jwa^lting mudh into their reports qn institutional 
history, financial status, and administratiye ade<?aacy. It is 
felt that there is a very -important role for the consurtant in 
evaluating thoroughly various business managemei^ aspects of ' 
the leveyof the program, project and/dr ceiiter. iJpelf . Hiis 
may mean dealing efJctensively with a number of ipicT-ievel manage- 
ment officials, but this is vlhere the real fcnd&ledge'dn the 
extent oT planning and prc^>osed operational) approaches can be 
obtainjed. It is also the level v*iere the -Consultant may te 
able to provide seme r^al assistance, to the ^rante^ staff in 
•making sugg^tions on how^ to dvercome identified inadequacies 
or prop»^ alternative approaches in areas v^ierie ne ittight ^ 
feel it apprdt>riate. . It is clear that there is- a need for 

• maximum benefit to be obtained frqn\thfe participation of 
qualified management personnel . riowever another misuse 
practiced by som^ of .the BII?s should be mentioned. From a 
review of surmary sheets/PSV reports and from the comMnts 
of t;he_ business management consultants themselves, it 
apparent their maximum efforts are, at ^ times ^ directed toward 
tracking, and .recording the multiple ttidgetary adjustments ^ 
mad^ in tft^ course of ^PSVjon the ^arge. grants. Si^h activity! 
v^ert corA)i^5d with the mbre responsible role, is adaptable? 
but, if i^rl considered -as the primary reason for having suqh 
individufli^particip^te in. the Feview, 'then this represents 

a' sfirious^Vuse of sudi expertise. - ' 
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P, THAT 6QSINESS MANAGEMBir OONSULTAHTS SHOULD CamOBtlTE TO 
OVER-ALL RBCXMfENDATIONS ON PRQJBCT SITE VISITS HtD AT 
IMinM. REVIEW GROUP^ MEETINGS, BOT TBE^ SBXID NOT VOTE 
ON "APPHDVAL V^ DISAPPROVAL* OR Gl^ A fRIORITY RftTING UNllSS, 
IN THE JUDGMEOT^OF THE IXBCOTIVE SBClRETARY, THE TYPE* OP PROJECT 
INCLUDES SIGNIFICANT NDN^SCIENTIFIC ASPECTS CLEARLY WITHIN 
THEIR PHQEESSION^ EXPERTISE. 



The question of vpting was a Very intriguing one. The business| 
management consultants themselves were some^t ^ocbivalent on 
the quest ion ;o£ Aether they shoilld' vote. There was an indi- 
cation that some abstain from voting even though they are 
almost routinely asked to do so. Hecognizfng that thf^peer 
. review is clearly an overall evaluation of scientific merit, 
an attempt was made to determine the basis on which the\ 
management <?xisultants woUld votS. It was feltvthat, if the 

M primarily, on their Sdministrative evaluation, 
questionable; if they were verting on v*iat they 
cons^tsus of the sci«itific reviewers' opinions, 
o be questionable, h*it for vastly different, 
jreasons. yas difficult to get any kind of uniform respcxise 
frcm the management consultants, so the question was also raised 
with, the Executive Secretaries and Jiie ccranittee diairroen. • 
Bare/ the question was mor^ difficult because they were bein^ 
asked ^sj^SI^SI^^^^ basis of the managemmt coraaaltant's 

vote.^^i^^^^^xDnses wiere «feven roorfe variable, but generally 
'led to d^i^ssion of the fact that it really did not matter 
because the management consultants represented, sudi a limited 
minority in the typicaCT voting situation that /their individual 
vote would have no in{>act anyway. Many of the chairmen 
reversed their previously stated position that the management 
, consultants s^kaiild have the vote wh^ asked how they would 
'\ f eer if the total participatj^. in the review activity involved 
» poly six or/s^en individual«f rather thai? 16 to 22. The 

sj^rong feeiing developed that, while the managemmt cohsul^t§ 
could provide significant input or findings %^rthy of serious 
consideration by the Whole group of sci^i^if r^eviewer* in 
Ipok^ at ^the total application package, the business consul- 
'tahtis. should^ not have the option to vote. An excQ)tion to this 
rule yfoald made , if the .Executive Secretary of the IRG jor 
PSV t^Bn makes detennlnat^on that the nature of the project 
^includes significant; non-scientific aspect^ clearly within the 
profess i(xial expd^ise of the business managemmt consultant. 
In t^ls case, the1|consultant coi^ exercise a full vote on 
approval v^ dis^3fpoval and on the priority score. 

^ G^. THAT BUSINESS 1|NAGQ(ENT OOtSUL^EANTS SHOUU) PREPARE A SPECIFIC 
FORHON OF PB^BT SITE VISIT REPCKTS AMDA>R IRG ^DWIARY 

smmEinis whi^ should be recognized as separate frcm scien-* 

TIFIC review and EVALUATION, ^ SHOULD QE CAREFUIIiY OONSII^RED 
IN REACHiNG RBOOWENDATIONS fCR EfCB PROJECT 
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Itiis reconmend^tion is closjBXy related in some ways to the . 
grgv ious on| on\ the "vote . * Again, recognizing thq basic^ , 
responsiBillty for scientific merit assessment by yie initial 
reviewers^ it felt that the ^siness roanaganent evaluation 
should be kept distinct from the scientific evaluaticMi^ The 
business mana^snent cbhsultant's portion ^ the r^x)rt could 



reviewers dt the- total program, but it should be clearly 
set^ate from the scientific critiques of program segments. ' 
Most of those individuals questioned agreed completely with 
this position. • . ^ 

H. mT PRDCEDUiffiS BE DEVELOT>ED WHICH WOUID ALLOW TOE REPORTS QE-^ 
^ ifiE^BUSDffiSS MANAC2MENT CONSULTANTS, INCLUDING SPECIFIC rIBSh- ^ 
. ME^iDATIONS FC»^ MANMXMENT IMPFDVEMEMT, TO BE MADE AVAILABLE 

TO TOE APPLICANT IIBTITUTION BY TOE KID AT AN APPRDP^RIATE^ 
^ TIME AFTER COMPLETION OF THE REVIEW. 

Ttiis recoiihendatior). relates to the rationale for E.l, 2,' 3, 
'and 4 i( content) and G (format) ^bovje. With appropriate guide- 
lines for the nianagement consultants, it is *felt that^nan^ 
inbre r^orts (than are currently seen) could provide signi- 
ficantly helpful advice for the roanc^eniait staff of the grantee * 
^titutions. If this is the qase, then that infonteti<m shcxild 
be readily transf er^le by staff of the awarding coirponent 
to a^ropriate. personnel at the-^institutions v^re it could 
be'utiliz^. Inplementation. of this reconinendation is not ^ 
now pr!^)sed, but rather it is suggested that a standardized 
report rorroating or other ideas should be considered to-roake 
this reconmendaticxi feasible. The grants management ^officer 
should be responsible for coordinating this activity within^--" 
the BID. • s • 

I. JlHA'T GRANTS MANWEMEWT OFFiCH^S "OF IHE BIDS PAKTtCIPATE WITH 
, THE IRG EXECUTIVE SECRETARIES iN\'fflE bETERMINATIOl OF A NEED 

R5R, AND IN SELECTION OF, ' BUSMESS MANJ«2MENr CONSULTANTS FOR 
THE .PROJECT SITE^VISIT TEAM Hs^LVEMENT . - 

Under ];^onTnendation ^, reference was^jnade to 'the fact that 
%^the need for business manageroen|^ c<xisultant support should be' 
related to the organizational or adroinistrati^e corrplexity of 
thor individual grant application. With the grants rfianag&nerit 
' off rcersV background experience and concerrZs^^or smooth pre- and-- 
post-award , administration, and the fa^t that they sometimes have 
different ways of looking at issuee-^rom program managat^it 
staff), it seens reasonable ^and logical 'to propose their 
'participation with the. Executive Secretaries ii> making the 
determination of heed. Likewise, because of their ccxitinuing 
day-to-^y involvement^ with business management staffs of 
grantee instituticxis, it was felt that they could be particu- . 
larly helpful in flection of the business management cc^ultants. 



be critical to the overall 




\ r \ ^ 

3. IHAT, WHQEVHl POSSIBLE, BID GRANTS MjkftffiMBlT STAFF MEMBERS 
SHCXJLD ACX3QMPANY TOE SITE VISIT TEAMS ON TOE REVIEWS OF TOEl. 
COMPLEX PROJECTS, SERVING AS STAFF RESOURCES ONLY. 

This reconinaxlation caroe primarily fran discussions with 
the Executive Secretaries ^ the IRG/PSV diainnen. Some 
feeling was expressed that grants manageroCTit staff could serve 
in lieu of business management consultants, performing the 
same role, except that they would not be ccMisidered as 
actual mCTbers of the PSV team arid, of course, would not 
vot^. However, the major itiy did not favor this approach for 
a variety of reasonfe, not the least of whidi was related to 
9 possible conflict of interest. Here, an analogy was dr^ 
to the impropriety of utilizing an extramural BID progrffin 
manager as a scientific reviewer. Perh^>s a more practical 
.i^reason. f or negativity was based on the belief th^ the grants 
management staff were, in general, not experienced in grantee 
institution managemeit anci could not really -do the Sccne typ^t 
of job as the business management consultant ^o "knew- the * 
territory" and would "IBfe^able to "open necessary doors," 
While agreeing, it was felt that the basis for the above 
recommeidation is: (Ij/that the grants mc^iag«n^t^t5iff can 
and do serve as very yaluable staff resources because of their 
intimate knowledge of 'grants policies and procedures; and / 
(2) that the firsthand exposure to the institutional strff, 
facilities, and environment at the time of review woul»provide 
the grants management staff with many insights v^iich coild be 
of great value 4n^ carrying out later administrative respon- v 
sibilities in the evOTt, a grant i? awarded, ^ 

Conclusions 

In FY 1975, NIH obligated approximately $416 million for large 
^program project and center grants. It is clear that NIH-wide 
policies and proi^ures should be developed spe<;ifying business 
management tediniques to be employed in the grant peer review • 
process. At present, no such directive exist and BID' Executive 
Secretaries, business management ccmsultants, ah3 IRG or PSV 
diairmen have no uniform guidance in this inportant area, 
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THE RDLE OF PEER REVIEW IK SUPgQRT OF UNORIHODOX, INNOVATIVE RESEARCH 

In any process dealing with the support of research and development there 
roust be provision .the identification of unusual and innovative approadies.' 
Creative and urorthodox avenues of investigation may herald great^ leaps 
forward in concq?t; they may represent a turning point in research directions; 
or they mc^r all too often, be fruitless attenpts to overcome ctostacles tp - 
further progress. 

I. DISCUSSION 

Within NIH, as in the medical and scientific c9l*unity at large, there 
is concern that the peer review system favors the support of traditional, 
conservative, orthodojf research and that this is at the expense of 

* innovative, unorthodox research. This perception is further conplicated 
by the view held by many that innovative, especially creative research, 
is usually conceived by young investigators whose research ideas are 
usually untried, may not be well received and the significance of which 
not even be recognized*! Consequently, such research applications are 
frequently disapproved or, if approved, at a pridrity which is not 

. fundable. We are certaih that this course of events does occur but 
we do^^ot know how often and in v*iat circumsitances it is most likely 
to occikI Moreover, the Sfaidy'Team does not believe there is any 
ineluctable relationship between youth and creativity. 

One of the criteria used by IPGs in their scientific and technical 
merit review involves' an assessment of the probability of success of 
the proposed project. If the concept is truly innovative and if the 
approach to the problem is unorthodox, then the reviewers are con- 
fronted ^ith a dilonma.. A proposal that- involves an idea that is 
innovative or unorthodox requires an assessmoit of probable success 
and this may assume evenf|reater weight and significance* Often, 
the evidence indicating *^bable success is limited and, therefore, 

*many such applications are not reopmmended with enthusiasm. And 
yet,, most reviewers are truly eaget to find, identify, and recpnmend / 
support for fresh nev> ideas. If an application contains a new or 
ev«i unconventional approacii to a problon and if it is well conceivfd 
ip its conponent elements, IHSs will generally recoimend support 
' -^^ven though they believe, that the likelihood of probable success may 

^ be niarg^tnaloJEndeed, this eagerness may lead to a generous assessment ; 
of probable su^es& -in-order to assure^ favorable recommendation for 
support. If, however, the' pEopc^l is poorly conceived and/or inade- 
' quately prepared, IHGs are less likely-tamake a generous or favorable 
assessment of prc*ah^e sugcess of the project 7 "land^^liejjltimate* 
reconmendation may welHSe^disapproval. Ttius, in these c±f ca^t^nces , 
the applicant would seem to have a special responsibility to write 
clearly, explain fully, and, if possible, report the results or trends 
of pilot studies In short, to conpete successfully the innovative ^ 
and' unorthodox proposal should be buttressed by an especially'weil 
prepared application. . 




It^^cyld be noted that the^ NIH mechanisms of researdi .support represent 
imrestments designed to attain a l?etter understanding o£ di^fease ^ 
processes that may lead ultimately to inprovements in healtn. They do 
not* r^resait'^invesbnents in' individuals ooly# in people 2S£ se~ . 
even those who are ^xpecially creative and innovative. The investment 
is in people oaly to the extent, that they conceive research ideas 
that may lead tp the afore^nentioned results. - . • 

The membership of NIH peer review groups is- seen by sane as a force 
tending to preserve orthodoxy. Hie members are selected because of 
theft ccnpetence in research, their ^ntributions to biomedical know-^ 
ledge, their stature, and their willingness to devote long hours io- 
these^duous review tasks. Some believe that, by definition, they 
are "th^ establishment." Ihey represent, in the aggregate, the " 
curroit state of the art, the current vanguard of researdi — but not 
tomorrow's. These tendencies toward orthodoxy do indeed exijgt but 
they are 'of f set by several factors: the efforts to have IK3s reflect 
a broad range of researdi viewpoints; the appointro^t of constructive,^ 
critical # enthusiastic investigators to review groups; the deliberate ' 
att^npts to have outstanding young investigators on initial review 
groups; and the eager conpet it ion within review groups to detect and. 
recognize the innovative, unorthcWox proposal that may c^n new and 
promising avenues of investigation. ^ ^ 

Certainly, the most subtle and most pervptsive influence that fosters 
the support of safe, orthodox research, as opposed to the creative 
and innovative, is the limited availability of funds. As the dollars ^ 
available for research support become more and more limited, there 
is a tendency (almost inevitable and often unccMitrollable) to invest 
in the "safe bet." In such circumstances, advisors, consultants, . 
staff members, and program managers avoid the high risk, *the hon- 
ccHiformist, innovatjlve prc^^al^ Indeed, chronic or recurrent ^ 
shortages of funds may indicate, and may cause, the demise of a ttuly 
vital research program. • . - \ 

Because so many individuals asso9iate especially creative, imovative . 
research with new and/or young' investigators, the qu&d^on of peer 
review as it relates to the support of new investigators is relevant 
here. Data derived from the study conducted by Douglgss and James(l) 
indicate that, for the period from 19€6 to 1972, a{)proximately ten per- 
cent of all principal investigators, on research project grants were 
being supported by NIH for the first time. Unpublishjed data for the 
piriod, 1972 to 1975, indicate that the percentage has remained about . 
the same: about 53 percent of all new researdi project awards were 
received from investigators entering the NIH G0(ppetitive system for the 
first time. Furthermore^ in most .Institutes the success (i.e'., funding) 
rate for the younger investigators is better than that experienced by 



the older applicants, it should be noted, however,, that the most 
critical and. useful index for the success' of "the peer feview system 
is the qualirty of researdi that has been support^, over the yeHrs, 
the. advices ^SnSOt knowledge which have-tJome about from sudi 
'suppEirt. ■ • ' / / : ^ 

How can the especially innovative research>pr<^al Identified? . 
Ihoee-iAio sutmit vAiat they ,consideij to p^ creative, unorthodox 
research applications should be encourag[ed to identify their research- ; 
as innovative; they should be required to state clearly and to document , 

• fully the innovative aspects of their proposal and the s^ignificant 
ccaisequences that .may emerge if aJpport is provided. 

The Study Team is erf the opinion that the initial /ceview groups have 
a particular responsibility In the identification of this' type of 
inncjvative research'. This recjjires perception, reflection, and- the 
provision of adequate time* for discussion. It is not easy to accanplish 
in circumstances where a review grcftip is • overburdened and tired. It is 
convinced that^th^ IHG members are truly dedicated to such efforts. In 
fact, the/ seem to share "the joy of discovery v*ien they can identify and 
d^cribe-tshe creative insight, the new-'approadi, or, the truly 
innovative prc^)0sal. They .are meh and wonen of broad"' vision whp are 
anxious to ensure that the present system provides for *he adequate . •_ 
recognition and support of Innovative research. • , 

I ' - ' ' ■ 

teOOMfeMDATIONS A - 

r. TOAT NIH REQUIRE TOE ' aPPLICSNT TD' IDENHFY AND SUPPORT IN D^-A^J. . 
- .TOE CONTENTION THAT THE RESEARCH PROJECT BEING PROPOSED IS 
ESPECIALLY INNOVAHVE; . , 

2. THAT INITIAL REVIEW GROUPS BE REQUESTED TO- IDENTIFY APPLICATIONS 
•niEY CONSIDER TO Bt ESPECIMiY CREATIVE OR INNOVBfflVB, WHETHER 
OR NOT THE APPLICATIONS WERE SO IDENTIFIED. BY TJffi APPLICANT; ^ 

3 " THAT IBG MEMBERS (AS" A GROUP 'OR INDIVIDUALLY) BE ENgCOW^GED-'EA.' 
. PREPARE A STATBMERmj ADDITION TO THE RBGULA^l SUJtlARY STATE^EIT^ 
POINTING OUT THE INNOVATIVE ASPECTS bP THE APPLICATION AND ITS 
SIGNIFICANCE. THIS STATEMEKr--IS TO BET>REPARED REGARDLESS OF 

• WHETHER TOE APPLICATION. WAS RECOMMENDED FOg APP^DVAL OR DISAPPROVAL; 
• THESE APPLICATIONS WILL BE FLAGGED FOR SPECIAL OCNSIDERATICN BY 

BID STAFF AND/OR. ADVISC»Y COUNCIL; „ ■ ^ 

4.' THAT NIH CONSipER THE FEASIBILITY OF DEVELOPING AN EXPERIMENT 
' ■ INVOLVING LIMITED SUPPORT FOB CERTAIN SPECULATIVE, HKM^SK, 
' ' 'INNOVATIVE RESEARCH PROPOSALS. -SUCH A STUDlf-MIGHT BE PJWT OF A 

LARSER, MUCH-NEEDED EFFORT TO EXAMINE THE PFDCESSES -Of DECISION- . 
r MAKING IN ALLOCATING RESEARCH SUPPORT., ♦ 
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impact' of inappropriate DISCIitisURE OF REVIgW DELIBERATION 



dOal peer review process, and the decision-making process, have been 
^^scribed |:n pi:evious^^tions and will not be restated Jhere.» In addition, 
theuitpact of the Privacy Act on the peer r^iew pH^essT^s been discussed 
" • in detail. Ih^ impUcations *ajfi| ti)is Act are.th^ the principal investigator 
. has a'right^ to.a (xxnplete and detailed '^lahation of the review and 

decision concern in^tiis/h^r apt)l icat ion 7" However , this explanation must - 
be e^curate reine<^t the final criti^e as prepared by the Executive 
\, ie^retaryof the -particular IRG involved. # \ — - 

, 'Final formal action on an Sblicafeion* is taken'^only after*^ nurabei: of ste^ 
in^the jreview artd dteciaion-ilaking ^i^ess have occurred. Nc^o^e action 
albno^Sls or<||^ly pat^ can be corSsidered binding until the final Institute 
II jppsttionH^ been offically conveyed to the ^applicant- institution and 

' pRifKipal , investigator.. • ^ * v * 

; ^Unauthorised pranature d[isclosure^f the results of any portion of the total 
r^iei^ prpce^s may^' clause serious unnecessary problems . . Itius, .a potential 
grants cannot assi|me that an aw^d will or will not be made based, on ^a. 
* ^ rurtor or on casual ccqpfints by p^sons involved in the review process 6r 

by NIH .review or ^ administrative fprogram)^ta^f . Such coirmen€s are ^* . • 

\^ina]^rt3p.riate', and if matle or' if rumors are circulate^,' applicants do- 
■•^expect cert^in^ctiOTS to occur. For this reas6n, the proceedings ^of 
^previev^> meetings, the discussions whtch were held, the preliminary results 
^ of a feite' visit,- the written preliminary cbnments of the assigned * ' 
rev^iewers,"* are .all consid^ed ^ 'confidential working materials; they^must • 
^t^b^di«cussed cpenly by th6s€ involved in the review or by the staff, 
' with the ^ipplicant- investigator^ or ^itK any -qthec* unauthorized persons. ^ 

* * ' " t ^ ' ' \ 

.. It is obvious that there are many points during the process at vAich 

premature disclosure can'occur. For exanple, if a. project site visit- is 
" peffofined as part o£ a large review,- the report which is pr^epared for ' ^ 
pi^mission to a parent initial review group, is cleSrly preliminary sfhd ' . , 
mus^ be considered a working dpcument. -Th^e written. cotnftents ot assi^ed / 
reviewers 'and ouftside consultants, similarly* are preliminary working^^ 
(tocumaits. After- full discussion ajt an IPG meetihg, the essence ;0f the, 
roundtable dfscussior) arxi the salient features on which r^oAnerdations^ 
^' are based, are. incorporated i)^tothe summary statement prepar^ by the 

Exi^ti veneer etary. Again, itShould be emphasiz^ that t):5«.E^reliminary f. 
. evaluations may be signif icaptly changed as a. result o^^^^ditional informa- 
tion and discusson during'^bhe meeting, ai^ are not neceararlly' reflected 
' in the final/suntnary statement. ^Once'^the relevant porjzaons oL tJie .working 
' docini^nt^ JUndividual'. preliminary reviews, igite^yis'i^report^,- etc. ) 

haVe beenjfifeed for the preparation •of the off icialy^ummary report, they, are 
\* of no further use or signif icance^^xJ thus-shouidaiot be r^taine ^ 



Furthermore^-* it often happens that an Acjvisory CouncLl/Board may recarmend 
and an Institute may make', an^^ard for a grant which was recommerjded for 
^toval by the IPG witii only mAJerate enthusicism bcit which involves 
Studies witlfln an area of high prcc[ram. irtt^rest to the BID, CorlverselyV 
ajgrant applicatipn which has ^eeil approved with a Jii^h priority score 
may "have to w^it for funding if other program' priorities take precedence 
and "there, are limited funds avail^le, . Thps, even the summary statement^ 
prepared by IPGs do not reflect final BID action but only the IR3 ^ 
reconineridSt ion • , > ^ 

*It is therefore esstential^that confidentiality be maiAained by those, 
persons involved irr the review and the "B^inistratiVe details of the 
prt>cess so that the applicant- investigator obtains the appropriate and 
accurate information. ^ * 

Hie Study Team recarmend s /that NIH reit^ate its policy that all persons^ ^ 
involved in grants peer review be constantly alert concerning the need 
for %Dnf identialLty of all aspects of the review,^ and tTTat thene be*' 
strict 'adherence to the policy concerning the prohibition of prVmatul;e 
unofficial ^3iscldsure of the resultfs'of site visits,, IPG review, and 
Council actions, I 



RELEASE OF SUMMARY STATEMENTS TO PRINCIPAL INVESTIGATOBS 

NIH takes very ser iol^sly . its responsibility for informing principal \ 
investigators -and > their institutions about the outcOTie of grant application 
review. Over the years, N^EH procedures and practices in this area have 
^olved from (1) routinely providing only the "bottom line" {i;e.^ "approval, 
"disapproval^^ or "deferral") to principal investigators and their ^ 
iHgtitutions to (2) providing principal investigators, .upon request, with* 
in^ortarvt additional iriformation, hamely, paraphrased versions of the. 
reviewers' critiques of their appiications (the "reasons fetters"). , , 
Because many of* the dbmnents^and criticisms associated with a grant 
application -pertain ^ir^tly' to the qua/ity of the principal investigator's 
ideas, training, experiences, and accorplishments, NIH traditionally has 
viewed ^e reasons letter as a confidential communication to the investigator 
and has. left to* him/her^ the decision as to whether or not to share the letter 
with others. . . ' ' - - ^ ^ 

/ ' * > ' . 

In ojrder to make the best, use of the technical merA: assessments of grant 
apE^lications t^y peer ceviewers, NIH staff is considering the feasibility 
and desirability of adopting additional or alternative approaches for 
communicating with principal investigators about the results of review. « 
For example, some believe that the reasons letters should be sent ^ut 
routinely on the^ initiative of NIH rather than strictly upon request. 
Others note* that parapdirasing^ the contents of a summary statement produces,^ 
at best, only a good approxifna tip n of the reviewers* ccniDents and, at worst, 
an account that is. incomplete, • inaccutate, or otherwise distorted as 
compared to the original. Still others point out that the t^sk of 
prep^ing reasons ^ letters is a significant c^omponent of the^ workload of 
BID extramural/staff 

T^e Privatfy Act has added a new dimension of concern. As now -implemented, 
the BIDS provide edited suirmafy statenents, without priority scores, tb 
principal ^ investigators upon request, even if the request ; is ,feceivecT 
before a National Advisory Council/Board has corpletefl, act'ioii 'on the . 
c^jplicatiorf. -Therefote,. usid^r present NIH policies, an investigator who 
exercises his/her rights to thejlimit, may have an almost complete copy^ 
t'of the initial review group summ^ ^tatemen): concerning* bis/her application 
^even before a National Adv-isory '^^icil/Board ^has had the opportunity to 
consider it-. At ^he oth^r Qj^treme,' an investigator .who takes.no initiative 
beyond .submisson ^of an application may never learn anything 'm6re„ than the . . 
final ictlffc' taken by* a National Advisory Gouncil/Board. a ' 

Given the impgrtance that NIH attaches to' good cprrmunic^ions,. it does not 
seem in^the interests of eithec the principal investigators or^ the NIH Xo 
allow tihe " irtf CMTTiat ion contained in Sunmary statenerrts to exist only in Nrt 
files beqause investigator^ ptey not understand the lav and/or NIFT procedures 
well enough iioj^now how to obtain these detailed -critiques of their 
applicatibr^s. Because of 'th? foregoing concerns*/ the. Study Team explored 
the.c^QStion of whether NIH Shotild go beypnd -its present practice of 
inform ing'^ principal investigators ab&ut review results and,' if So, what new . 
.approadhes should be coctsidered. . - ^ m 
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REXX>fgH)ATIONS AND DlgCy 

1. .THAT, AS SQCW'AS PRACTICAL AFTER A BID'S NATICNAA lADVISQRY' COUNCIL/ 
BOARD COMPLETES THE REVIEW" OF' A GRANT APPLICATION, THE BID SHOULD 
' ROUTINELY. SEND THE SU^rtARY STATWENT, VflTH THE PRIORITY SCORE » * 
. DISPLAYED, TO THE PRINCIPAL INVESTIGATOR. BECAt^E SUWlAfiY STATE- 
•• MENTS'FREQOEmY CONTAIN FORTORIGHT AND CRITICAL C0»1E!frS CCtCERNING 
'THE PRINCIPAL INVESTIGAIORS/.-THIS R^XMffiNDATION IS MADE CONTINGENT ' 
UPOti THE UNDE^^ANdlNG THAT THESE= lXX::UMEmS WILL BE feELEASED BY NIH 
ONLY TO 'MESE, INDIVIDUALS.,. IF IT lS<dpil3WINED THAT SUCH'NIH- . - 
INITIATED REL^E OP SUMM^VRY -STATEMENIS COULD REASCWABLY. LEAD TO - 
A REQUIREMENT TOAT 'nIH M^E T^ESE DOCUMEOTS AVAILABLE ALSO"TO 
' APPLICANT INSTITUTICWS OR' OTHER$-^ THEN THE'STlBY TEAM RECOfrWENDS 

THAT *SUJ*IARY STATEMENTS "WITH PRIORITY "SCORES DISPLAYED BE- - ' , ' 
RELEASED ONLY UNDER PRIVACY ACT PROCEDURES, I.E., UPC»J RECJeIIT'. 
OF AN APPRCPRIATE REQUEST FROM THE • INDIVIDUAL CCWGERNH). \ ' ' 

Itlis recommendation was one of th^ most difficult for the Study ♦ 
Team to forraulate and was arrived at only after agonizing delit>eration,' 
.On the one hand, t^e Study Team is uhanimous in believing that , 
principal investigators "deserve tchave a detailed account of the 
results of review in a timely fashion and, in a form thi.ab 4s mbst . * 
likely to be a -ccpiplete ,and accurate account of^ the reviewers', , 
^ i findings and recommendations, ,0n the other hand, tne , Study Team 
is unaniiTK)us in believing that tilH should take no action that 
would^ jeopardize oi^endanger the privacy rights of individuals. 
The Study Team would bei deeply (distressed if NIIH init iated release 
^ ' !^bf ^summary statejnents to pi: inc ipal itwestigSitors somehow were to ^ 

' trigger or catalyze aychain Of events (e.g., judicial act ions 'or 
, legislative reinterpretations) that caused the provisicMis. of the 

Privacy Act no longer to apply to summary statenjents. ^Because pT 
* this ambivalence the Study Team elected to laake its recomnendation 
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'conditional and urges' that NiH seek gu 
earliest po£sjJ9.e tiuie. 



d to ma 
ihanc^ 



on this matter at the 



Carefully formulated* .it ^ ft, ►the .r^qoj|jtodation ^Aill engendered ' 
a significant diversity d# reactions fdB study featt^ members, , / r 
Some^ regarded it as the strong^t- reco|Wen|fetion possible in the 
\,face of uncertainties'^ about judicial ahd/oF legal -.conseqiwnces' 
: and therefore, ifdorsed it enthu^astically. i^^he^s feared that. 
NIH^^itiated mailing 'Of surtnary stat^ement^"^ ri5asons|Letters;- \ 
toprm^ipay^inve$tigators would b^ interpreted 'by the scientific 
community<^$ gratuitous comniunicatiory5''or , even worse^as attempts 
to d i] ict T^earch|t Consequently; th^ preferred ^the present 
pract :c% of providing detailed accounts 6t rfevi^^^^^ulrbs only 
upon lequest. This latter group, witfiin the SrtSy Team, believed 
that me inefficiency associated with havinf principal investigators 
. . explicitly request summary state^oits or reasons letters is an • * 
. acceptable price to pay for ensurir^' ^at (Jetted Mcounts of , 
the result^ of revie# go onJ^jJto €hose prinoipai imw&ptigators w^io 
' take specific act ion, to obtain than. 
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Although routirie release of sunmary statements to principaj investi- 
gators would be an NIH initiative rather than the investigator's, 
the sunmary statein^ts would nevertheless be processed consistent 
Inth current NIH Privacy Act procedures. This involves having' them 
edited as necessary zo ensure ti>at the principal ^^pvestigator- 
receives cmly that informjption which pertains to'hjjn/h^. While, 
inplenientation o'f such a procedure clearly is fteasibleyMt will 
produce sane increase in the workload of eiD officials thoughout<j 

>NIH/ especially during the weeks iirmediat^ly following 'National J 
Advisory Council/toard meetings. ' j ^ ' ! » 

.»The bioroedicdl fese^reh community is becoming incre^ingly ktxwledge- 

• able abouti the rights of individuals under' th€f Privacy* Act, and 

" seveial BIDs are r^XDrting an increase in the number of requ^ts 

from principal investigators for the ,;5iminacy statements concerning 
^Vtheir applications, therefore, NIH-initijited release pf summary 

rstateroents^ife likely only to hasten ^ growth in workload that ^ 
already is inevitable and shift it from o^icial^ S^ignated to * , 

* ;haidle ^ivacy Act/ procedures to other BID officials. ' • 

The use of thWsunmary statement as a v^icle- for communication from 

• ^Nlft to principal invest igati)rs (Aether at NIH'.s initiative or ugon 
A:4^e9t) may present, another problem. For thctee appliCatioiis on 

\hich a Natioftitl AdvisoryJ^ouneil/BO^d' acts itr concurrence with the 
, initial review groups* recommendat i<5n$ (the vast ^majority ^ of ca^eg), 

it, generally wiU be' sufficient to mail' only the sbmmary statements. , 
'flbwever, ..for those applications on v^ich a National Ady^i^ry Gcxmcil/ 

Bdard talcfes an artion that is^ 'different frbm or in addition to tljat ^ 
^ recommended by an initial review group, the BID would need tp - 

* incite,, with the summary statement, an indication of the Cc^trciVs/ 
Board's decision and the supporting- rationale. Case-by-case attention 
My senio2^BID staff thereiorfe will be necesari[. On the other hand, 
usLing surrnary statepen£s to "communicate with 'principal investigators 

^wcXald reduce the workload, of BIft, extramural st§iff ihsdfar a$ the 
prep|iration of reasons letters^ is conceded. • *4 

' Dfespite these predictable diftfA^ties associated 'with in^^^ementing 
its reccmmendatioo# the Study ream believes that the' advantages of 
, ^ making Nummary statements read il-y\ availably to principal investigators^ 
' (hopejuljfy at NiH's initiative) fair outweigh the iS^isadvantages . ^ 
First, ihvestigators would have mpre direct knowle3ge of the a^or.qjuhr 

, nes«" a«3 even-handi^ness with which peer reviewers carl-y out tii'eir . , 
tasks; this aloije cjpuld dp much, tc reduee concerns ^Jpout ,the fffeclive- 
ness al^ equity bf . this process. Second^ principal* invest iciator*?^ 
WQuld l^e tb^ Denef it of valuable*" advice and .consultation from^a . " 
group ofjcpie^ledgeable and distinguished experjs.. ' Third, the _ 
khcrWilSd^ethat summary sfeatemefits'ar*, laltimatel^^ to-'be avail ^le ' 
oulbside, NIH as%welLas within; wbu]^ be an4pc'^ti^^e for^both \(tkr\\ . 

. 'v*b j^repare then and, those who use than in ded: isj^on-^nak ing to bP 
<»rtaiT\. that. fti^ if* contents are accurate^^^fompiete, ancl readable, ani 
tAftt references to speci^c investigators aremaci^wit^ full respect • 



>/ • r " , ^ / * 

for 't/ie dignity Of .the individuals involved: Fourth, ^;if^suIn^ary' 
stat|roents were* to be released with the priority scores displayed, 
investigators woii^d have a much clearer picture of how^eviewers 
perceived the relative scientific merit of , their applications and 
.>?CHild be .in a better position t© appreciate' how -program relev^npe^. 
oons^ifleii^ions and/or BID staff decisions effect their chanc^pNfeor } 
ah' award, .Fifth, NIH 'would be responsive to the ' ad^Hce^f the ^ 
scientific and lay public, for the condept of transmitting ^ ' , - 
-summary statements to principal ^investigators eithilr autanatically 
Of ipon * request , received a relatively strong ^endorsement frcwn both, 
tl^^participants in the NIH grants peer review/system'^upplenent^l) 
and 'Spme members of *the^ public ^both- scientists and lay p^Xe) who 
.testified at the- public hearing^* . * ^ . '/ 
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Although the availability of suirmary statements, with prior jrty seorfes • 
displayed, t6 principal investigators, jwill -provide them with potepti^ly 
. useful information about the scientific merit revi^>( of' their applica-- 
-tions, communicetions r yarding the likelihood of awarding of ^jp^ovfed 
^ applications are inadequate. It is not generally appreciated that, in 
addition to the scient*ific merit evalu'^tlon, subsequent decisions 
concerr^ng award. of approved applications are made at the PID level*, 
'\bised on prograimat^c and/c(r budgetary considerations. An accurate 
and cjear explanation of" thfese important additional considerations * 
is essential to avoid misunderstant3ings and unjustified assuppt ions, 
on the part of unfunded investigators, regarding the quality fef the 
scientific merit review of their applications; such misunderstandings 
frequently result in unjustified complaints aix)ut the peer ir^iew ^ - 
system. . ' % . • ^ 

^ THAJ^ 'WHEN^,' FOUmm^ COMPLETIQN OF THE INITIAL TECHNICAL MERIT 
'REvlfeW OF: A GRANT APPLICvATION BUT PRIOJR* TO FINAL AGTICM^ BY A ^ ' 
.NAJIONAL ADVISORY COUNClL/BOARD, A BID COMPLlE^ WLTti A PRINClPAf. 
INVESTIGATOR^S REQUEST FOR A CXJPY OF THE PEmNENOl SUMMARY £TArE7, ' . 
*MpNT (AS PRESENT NIH POLICY FOR* IMPLEMENTING THE PRIV70 ACT. . • ' 
REQUIRES) , THE Bfo SI^tJLD PRO^DE THE DOCWffiNT WITH PRIORITY SCOI^ 
DISPLAYED AND ADVISE THE REQUESTOR THAT THE INFORMATI(» B^flNG , ; , . 
TRANSMUTED IS. INTERIM -IN NAIUBE ^AND THAT ANY ATTEMPT, $T THIS^BOINT ^ 
IN ?JiE BiVIEW PR0€ES4r TO MODIFY THE ORIGINM^-APPMp^IOl* OR PKNVffE' , j 
COMME>frARY fOR- O^SIDERATION BY THE CmflClL/BOARnl^Y RESULT IN. A ^ 
DflfERRAL OF THE APPLICATION TQ^rTHE l^E^ REV][EW CYCLE. - ' 

I^^^^ Study* Team believes that, ^in coftif^lying witiPrequieffts for summary 
stateiTients under the Privacy T^dt before ccxt^letion of the review ' ^ 
• process, BIDs should follow the same practice as is proposed in 
" Re<^prflmendatHon 1 above,, i.e., release the dociment with ,tQ[e priority 4 
scohe i^isp]^yed. There seeiws to be nt> compelling reason vAy the 
priority score should be 'Withheld in one case. but made available in 
anoth'er.. * However, BIDs should 'make it clear to requestors .that^ ' 
tliey Bre^ receiving inter i^n, information and that any attempt to - ^ 
ihfruence the outcdm? of *the review process, ^at this point, could* 
, produce.significajTt delay.* ' 1 . . ^ ' / 



THAT WLH SjHOULb R^EST AUTHORIZrfTlQN, THROUGH; EITHER RBGULATI(» 
OR.LBGISLATiON, AS APPRK>RIATE, TO REIiEASE AN INITIAL m/I^ GRCXJP . 
SUIWARY STATEMENT TO THE PRINCIPAL INVESTIGATOR NAMED IN THE ^ 
APPUGATION ONLY AFTER .TOE REVIEW OF HIS/HER GRANT APPLICATIW IS 
COMPLETE, i*e* AFTER REVIEW BY ^lATIONAL ADVISORt COUNCIL/BOARD. 

Under the preset interpretation of the Privacf Act (as desci^ibed 
above),'. prin(j^>al inv^estigators have the opportunity to obtain 
-interim information about the review of their applications and to# 
re^t to it, e.g. / by ^submitting adSitioKil material an3/or rebuttal 
statements. Ignoring the fact that such action roost often will 
prcducp a'delay, *in the review, the written ifiterchange between 
a pri|^ipal investigator and NIH could result in a more accur^t^^ 
assesH^it of th^ scifntific merit'bf the application and the 
investigator's *abil it tes. In fact, several participants in th6. 
public hearings stressed the desirability of incorporating a — y 
Rebuttal step into the grants peer review- proceBilre. i 

■ V ' t ■[ ■ 

On balance, hdwev^r, the Study Team believes that th^, release of 



to fund anyway, and sudi reactJlJnto premature inf onftat ion jnay 
serve -no -good purpose. There will be other occasicms when an 
application, judged to be highly meritorious by an .i57itial review 
group,, is not selected fof an Sward by the BID because a ftlational 
Aclvisory Council/Board assigned a "low program relevancje" rating; 
raising false hopes irf an investigator with, interim information 
• only to dash then irt a few months seems, most unfortunate. -The 
_JSJtiK3y Team believes that the^pubUc_inter est would be,;b^tter serv^ 
-if NIH were allpwed .to conpl^te'its peer review procedures befor'e 
sending jprincipaX Investigators an account of the reviewers* 
findingsiand recomnendations. Moreover, if t^IH^s policies were 
chari?^ such that summary 'Statements wer6 senjt out automatically * 
at the Qorr^^tioh of the review process, .the denial of reque^^s for* 
interim* docunentation would not affect what a requestor receive^'* 
but pnly v^en he/^e receives it and thus still woq^ld 1^ consisteYTt 
with the spjrit of the Privacy Act. ^ ^ , 




PRIORITY SOORgS ON SUMMARY STATEMEWIS. 



TraditiOTally, NIH has assigned to each fi^roved^raot application, a 
numerical scote signtfyirtg its relative scientific infer it as perceived by 
an initial , review group. Ttiis so-called priority score originally was 
determined by having each reviewer rate the application. on' a scale from 
1.0 (the highest) to 5.0 (the lowest) in , increments of 0.5/ averaging the 
individual ratings, and multipying by iOO. Hie resulting three-digit 
ni2±)er qpe to be called .the raw priority scofe and^ fpr i!any yeairB, was 
used uniformly throjjghout NIH. 



J * 
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But serious doubts about the process eventually developed. For exc 
cbncern arose in several quarters of NIH that, consciously -or otherwise, 
merrbers of initial revi^' gro\jps sometimes m.ight be assigning especially • 
good priority scores to grant applications in certain areas of special^ - 
interest to th^, in the hope of * increasing the likelihood that awards 
-^AXild be made. Similarly, some began to believe that' reviewers sometimes 
^ express their. lack of enthusiasm for certain areas of science by assigning 
comparatively low priority scores to* all applications falling wittiin tlpSe 
categories. /Obviously, to* the extent that review groups behave iri these 
%^ays, the utility 'of the raw priority score as a true .indicator of ' i 
r.elatiye scientific merit ger j^, would be compromised. • ^ ' 

In order to circumvent this, a mathematical procedure was devised to cornpfire 
and adjust all the scores assigned at.a*given meeting of an initial review 

Jgroop in concordance with those ratings assigned at, the group's, two preceding 
meetings, ' The" purpose Of this procedure* is to- place the applications • 

^•appraised by any one initial review group on i^noirmal distribution curve. 
Itius, if an unusual number of* applicat ions wefiP to be given scores_bigher 
gr lower than the "normal'^ valife- ( abduT ii5U ) , ' such soorefe* would be iredigtri-- 
buted OR the-curve' so that most of than fall in 'the middle "nqrmal"SttFea 
where, in theory at least, they should fall,. By this procedure, the mean 

'score* is adjusted to 250 and the standard deviation to 70. Th^ re'feoltant 
^ues ^re, -called the "normalized scopes." 'nie as^umpt;lort underlying thi^ . 
statistical, procedure is that, on the aveii^e, approved |ipplicationfe from 
th6 various NIH initial review groups are roughly. equivalent in terms of 
scientific merit. . ' j . ) 

Currently, there appear on summary statements both the* raw priority scores 
and the -normalized ones (if the priority scores of 25 or more applications - 
are available . to be* included in" the normalization procedure),. Each / 
'awarding unit 'has the opt ion' of using either system but' is consistent in 
this regard.' Confij^iop about the' procedure can and does occur, how6vef, ' 
particularly on th'e j^rt of member s^f National Advi^ry Councils/poafds. 
Moreover, the normalization process can Change the priority scor* signifi- 
cantly (on occasion, by 50 points), eSE>ecially if riviewers tronsaioiisly 
attCTpt to; manipulate the procedure. }/hen an ^justment of this Inagnitude^ 
is involved^, it could easily mean tJie difference between award and non-award 
of a grant depending upon which score is used by the BID in makihg'the #^ 
awarding decisions. Because of Jits concerns about these •fhatters, the Study 
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Team elected to review the pj:es«it practices involving th^ raw and normalized 
priority. scQtes and to attenpt to detetmine whether any dianges might be • 
indicated. ' - , " 

PECXMfflNDATIONS AND DISCUSSION^ ' - ^ 

1. THAT A "SINGLE fblORITY SOM^!" CDNVENTION SHOULQ BE ADOPTED FOR USE 
THBblXaJOUT NIH. ^Jh . 

\ Uie present practic^f having both raw and normalized priority scores 
-on sunmary statenents* and, leavijig to .e<aa:h Bip, the decision as .to which 
convention it will follow, otjviously, is viable and could be continued 
indefinitely, without irrepar&le harm to the grants peer review syst^. 
• * In the view of soroe NiH staff, however, it is becoming increasingly 
difficult to justify to thanselves, to tihe scientific community, and 
to others interested in NIH's activities, ,why it is appropriate for 
something as fundamental as the converition for scoring the perceived 
scientific met it of a grant appligatioA to vary from one BID to 
* another% Recognizirig the significance v^ich all BIDs attach to 
"priprity scores when they make award selections and other program 
management ^decisions, the Study Team believes thsft; a single. priority 
c'onvention for ^all of NIH would be preferable to th^ present 




ADOPTING A SINGLE PRIORITY -SCORE NOTATION SYSTEM^ FOR USE^ 
BY ^ BIDS^ TOE NIH SHOULD CONDUCT A^STODY OF BID PRACTICE S RE GARDING ' 
'JJTHE USE OF IHE "RAW" AND "NORMALIZED" PRIORITY SCORES, TO DETERMINE 
WHETHER THE UNIFORM NIH-WIDE CONVEt^hON SHOULD BE THE USE OF IHE RAW. 
SCORE EXCLUSIVELY OR TOE PRESENT NORMALIZED SCORES, WHENEVER THEY ARE 
AVAILABLE {OR OTHERWISE TOE RAW SCORES)^. OR TOE DEVELOPMENT OF A NEW 
PROCEDURE FOR COMPUTING, REPRESENTING, AND/CM^ J^DJUSTING PRIORITY 
SCORES TO COMPENSATE^ FOR OIFFfiRENCES IN GROUP RATING BEHAVIOR. 

. ^ i / • ' - ^ ^ 

There is considerable difference of opinion among the NIH awarding 
urrftBj^to the relative utilities of the "raw" and "normalized" 
priority scores in their program manageiJent activities. Some BIDs 
use the caw scores exclusively in the belief that, at least for 
.the initial xeview groups which serve their programs,' the normaliza- . 
tion procedaip^ does more to confound th^ interpretatipn o£ ^ ^ 

scientific merit -assessments th^ui to reduciBkriarice in ^e rating 
behavior of initial i:Wiew .gtgups, eitherb^p^een such groups or 
within the same group <5ver time.. Other 'bIDs take the opposite 
view'. Because the issue is sp important and the answer is not 
obvious, the decision to adopt a* single priority score convention 
should not be taken until th^re has been a study of the relative 
strengths and weaknesses of the two alternative practices presently 
in use. Moreover, the §tudy Team recognizes that there also are other 
potentially useful methods for .corrputing and/or mathematically 
adjusting- scientific merit^ratings and these also should be studied; 
For e9campl^v 'it has be^ suggested ' that t^e present thr^e-digit 
representation for priority scores may imply a greater degree of ^ 
* ' \ 



precision in tte initial review groups* ratings than they are able 
to acihieve and thalt a more course representation* (with or without 

• roatheroatica't adjustment to compensate for differences in group 
rating bdiavior) might be more appropriate. * ' . . 

r • • . 

3. THAT, UNTIL SUCH TIME AS A SINGLE NIH-WIDE PRIORITY SCORE NOTATIOJ 
SYSTEM IS ADOTTED, ALL C05>IES OF SUMMARY STATEMEMTg WHICH A GIVEN 
BID^SEIffiS TO' PRINCIPAL INVESTIGATORS SHOULb DISPLAY EITHER THE* RAW 
PRIORITY SCORE OR 1HE NOI^IALIZED PRIORITY SCORE (IF AVAILABLE) BUT 
' NOT BOTH, BEPENDING UPON \^HICH OF THE T^K) CCWVENTIC»« THE PARTICULAR 
BID POLLGMB. ' * , # 

The display of dual priority scores^ on the cities .df summary state- 
ments \rfiich are released to principal investigators wouW inevitably . 
^be a continuing source of difficulty- for BID staff and others as. ' ^ 
they attCTpt to advise principal investigators about the outcome of 
the review process and their prospects for receiving an award. 
Hiis difficulty can be avoided easily by instructing the staff of 
each BID to delete from the copies of sunmary statements which are 
to be sent to principal investigators' the Rfiority score that the 

• BID does not use in its decision making. Ihis special editing 
step would be needed of Course, only until the NIH ^^ts a 

s ingle ,priorit>U:SPore systan and the printed summarystatCTients 
then have only on^ score displayed. , * 

4i THAT, NIH SHOULD CONDUCT STUDIES OF: 

a. - VARIATIONS 'iti INPIVIDUAL REVIEWER AND REVIEW GROUP BEHAVIOR 

IN RATING APPLICATIONS,' OVER TIME, AND AMONG DIFFERENT IRGS, 
WJD OF TOE. FACfORS WHICH ACT TO INCREASE OR DECREASE SUO! 
VARIABIJilTY; , ' . 

t ' ' . " ■ 

b. VARIATIONS IN THE QUALITY OF GRANT APPLICATfDNS ASSIGNED TO 
. -A iSIVEN IRG FROM OHE REVIEW ROUND TO TOE NEXt,. OVER TIME; ' 

AND VARIATIONS IN TOE QUALITY OF GRANT APPLICATIONS ASSIGNED , 
TO DIFFERENT ipGS; ' ■ , 

c:~-THE EFFECTS ON THE REVIEW PIOCESS OF DISPLAYING. THE RAvT 

PRIORITY SCORE TO TOE INITIAL REVIEW GROUP MglBERS IMtolATELY 
AFTER TOE^ ASSIQI THEIR INDIVHOAL SCIEOTjfccV^RIT RATINGS,, 
AND OF GIVING TIffiM TOE-CPTI^, AT IHAT J^Mltf TO RBOPHfT 
DISCUSSON AND BE^TE Tlftl, APPLICAT^* 

The "nomuU-izing cotrputation used by-Nll ilS an^^^^^^^^w^pdjyre for 
compeiisating for variability in IRG*VFiti'ng-^3i^^avi6r over-ti 
Stud-ies a-* auid b. above wpuld attemptAp darerinine whetKer there 
are more direc^~approaches, such as "tfe gflfcif ic procedure suggested 

• for triA in £. , whichjnighf be used p cdl|unctlftn with,' or as a 
replacement for, the nArmalizing proq(fedure to assure consistent, 

' objectivQ^ IRG ratings Over time. Study b.- would be a sj^udy to 
'determine the^ektent to v^ich d if fief encei^ among IRGs or within an 
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IHG overt iinei in tjie ^average prior ity^^cpre and aj3proval/&isapf)roval 
ratio, reflect real differences in the quality^ of the applications * 
under rwiew xather than arbitrary shifts* in group rating, behavior. 
For study blr^etrospetftive studies using citation methodology or 
the like, might, be. appropriate.^ 

Under present initial rgview group prpcedures, ^h meniber ulti- , 
matply expresses his/her view of the scientific merit of a grant 
application which Is being recommended for approval bj j?r;ivately .y * 
assigning a numeric rating, Itiese individual ratingsjare collectedV_ 
by the executive, secretary and his/her stalE£^ ahd averted. ^^Because, 
t^e averaging of individi^al reviewers' ratings for any given y 
anpl i raf ^i ^ j ^ j^n^ral 1 v is not performed until well after the \ ^ 
conpletion of the group'« deliberations (oftep several days after ^ 
tthe meeting), reviewers have no automatic feedback afe tp either 
What specific rating eAch member lass igned o^ what the group's 
aggregate rating^was,' • 

Itiere are several problens* inherent in this' procedure that, in the' 
opinion of the Study Team, make a special study — and^sdme experimen- 
tation ^ith alternative applroaches — seem warranted. First, because* 
the groupvhas only the primary and secondary reviewers^ written . ^ 
compents^and the subsequent or af exchange of opinion^ av^ilad^e at 
the time of rating, the Itonbers may not.be fully aware of the nefture 
or extent of differences of opinion which may gxist within th§ group;' 
and the Chairman may inadvertently terminate discussion Wften, in 
fact, further deliberation irtght bfe likely- to produce a greater 
consensus/ Second, without access to the resultant raw priority ; , 
score,' umediately following its djlscussion anc3 rating of an ^ 
application, the group has no opportunity to satisfy* itself that 
the score is^fuly consistent with the spirit of its discuss ions 
and, if not, to reopen deliberations and possilqJ.y reflllje^he 
application. Under the present procedure It sometimes •happens^*' 
that the specif io wri^en and oral coitments frtan which the - ' 

executive secre^ry prepares the suimiary statement for an application 
are appreciably ^t variance (either more or less favorable in tone) • 
with -tte implications of the priority score actually assigned". , 

Itiefe are a variety of techniques to effect quick feedback within 
initial review groups ab6ut individual reviewerV ^ratings and/or 
the aggregate score. These techniques range from the display of a 
hand-held score by each reviewer (as in done* by judges of ^diving 
competitions), to the u?4 of sophisticated electronic syatqjg for 
instantgji^s recording of individual ratings, computet iohs pf- the 
aggregate score, and display of results. 'Ttie Study, Team recoifinends 
that NIH conduct a study comparing the present "no feedback" procedure 
with one or more alternatives involving immediate feedback about 
individual and/or group rating behavior. Sud) a §tudy should include 
mechanisins tp preserve the privacy of individual ratings by. IPG 
nlembers and should not. be construed as a means *to pressure IRG members 
to change their ratings. Ratljer, the studyv should be considered as a 
further means of -studying rating behavior and an effort to develop ah 
improyed rating proj:edure. 



In his J^ril* 28/19^75, memorandum establishing? the NIH Gra^itS Peer Review 
Study Team (GPRST), the Acting Director^, m% asked the Study Team to 
address 'the entire 'process of peer review," and tp give specific attejjtion 
to... •'the laeed, advantages * and disadvantages for a tedinjs^l merit reviey 
•appeal^' mechanism for applicants and an assessment of^>dfe proces^^currently 
in use.,.-'* This suggestion led to, formation of a Subcommittee on Appeals 
early in the activities of the GPRST. Iir addition to those members xiios^n 
from tt^ Study" Team, )the Subcommittee was supplemented by additional m^nber^' 
r^resenfcing various[ functions infthe grants :peec^ review prooess in different 
BIDS. ' Itie substancrf of the^eport of the Subcommittee was adop^ by the 
Grants Peer RevieJW l^ydy^^t^am. • . ' ' , ^ ' 

I. BACKfepOOND J * • * . . ^ • , 

Several factorlgiiave led to concerns' abcxat an NIH grants peer review 
'^)peals proceste, most notably, perhaps, a portion of the Report of 
the House of Repre^Hjgjatives Gg^renittee on Inter^tat^ Foreign 
Conmerce'on the NatlHal Cancer Amendments^of 1914. \ln that portion 
of the Report dealing with Peer Review of Graitt Applications and 
^ Contract Projects, the Ctmnittee expressed ^the hcpfe th^t NIH would 
take steos to ensure "...a just and proper recourse. • .to every . 
unsuccessful applicant who wishes to appeal an adverse decision,,", 
and also voiced the feeling ";..that,. if, the present appeals mecha- i 
rtisms are determined to.be insufficient, it would be appropriate to 
cotipider the creation of^art independent appeals process..." tl).- ^- 

, Other, action in tl^ 94th Congress focused more specifically on. an 
appeals system for the Natij.on^l Science Foundation. ^«ate Bill 
'S.. 242? and- House Bill H. R. 9492 both' suggested- t^aFNSF "...establish"- 
provisions for appellate procedures ^independently review... 
T)r<^)Qsals disapproyed^by the Foundatlwi.^. . " (Appendix *I-lh -While / 
these two bills failed to tfe enacted, they presumably hacj some 
' effect on. the NSF directorate, which issued, - in January 1976 an 
^ "^Important 'Notice on Reconsideration: of Proposals Declined by NSF ' • 
m {Appendix 1-2). This Notice aimsjto standardize th? agency's methods 
\ \ for reconsideration of 'pr'oposals which have been "declined;''' a tierm * 
, signifying that NSF will provide grant sqpport ih respchse to 
> an app]^ication. V 

Most specific ♦to NIH. and the Grants^ .P^ Review Study T^eam tasks, - 
a number of inputs to the GPRST have related to the need fiof an ^ 
improved NIH_^grants' review appeals systeja. Such opinions have" been 
expr^^ed in the three hearings held by the GPRST, have been r reived 
^ in a significant number of letters f rom* thfe biomedical researqh 

• coirtnunity, and have»appearfed in comments on the ^estionnaipe f rem. ^ 
mentoers' of NIH peer i review xxrni it tees^. All of these, comnents vEocus * 
on two broad aspects of an appeals system: ^ - - . 



0 need to establish a system, v^ereby. an applicant may 

challenge the assigrmierrt^^of* an^applicatiorl to.an initial . • 
review group (IPG) for evaluation of Scientific merit, and 
to a* Bureau, Institute, or Division {BID) for 'funding 

• TOHSideriations. Applicants have. beei> concerned regarding 
, difficulties ^th v^at they perceive as inappropriate 

assignn»its to'lRGs which would allow an application to.be 
evaluated by reviewers v*iq may not be the best qualified to 
do spi or whose members, might .have potential biases in relation 
to the applicant institution, the principal investigator or 
staff, §5r the suggested projedt. .Applicants sometimes also 
question -the, assignment to aa awarding unit^ or BID, which^ 
may haVe less potential interest in the proposed project 
than another BID. , * , • ^ 

0 The peed for a systan.vrtiich will en^le an applicant- investi- 
gator to receive the total information vA\ich was utilized in 
.making final reconmendations on. the application, and to have 
•the c^jportunity to challenge or rebut -such reccrtmendations. 
ApplicaAt-investigators sometimes percejv^ factual errors in 

• review, e.g.., .data or information overlooked or not considered 
in the review, or data or stat^nts misinterpreted by review [ 
groups. Or^they may 'question the validity of judgments 
regarding the scientific evaluation, -bi^et adjustments; or 
ethical' or bi(*azard considerations of proposed projects. 

Current Appeals Mechaiisms (-Appendix 1-3) ^ • . ' 

The present NIH grants peer review systdJn includes lorjg-established 
and well -tested- mechanisms for handling these processes regarding 
v^ich suggestions have been made for improved handLing of disputes 
or appeals. These involve the referral process, and 'the processes 
for coimiunications regarding the outcome of "scientific reviews, 
described elsewhere in this report. 

1. Referral , — The receipt and assignment of applications involve* 
several steps which are frucial and have important Implications 
for the application as it moves |Jirough the poer review system. 
These functions are responsibilities of the Referral Branch. in 
the Division of Researdi Grants ^DRG) , v^ose professional staff 
is selected on the basis of varied and specialized Backgrounds 
suitable for determining the c^^mpetencies of IPGs to revfew i 
' various appliCatiorjs^ and for iSentifying appropriate granting 
components in light "bfi their (established •areas of program 
interest. The primary consideration for IRG Assignment is that 
thfi^cienti^ic methodology described in the application is 
best evaluated by eminently. qualified scientists who, themselves 
' use or are familiar with the same methodologi^; Assignment 

• to a BID derives. from an overall evaluation of the aims stated 

^ irv the- application and the investigator ''S statement of the ; ^ 
sjygnificarice'of the proposed work. -Actual assignments are / 
based primarily on gQidelines developed by DRG in collaboration 
with th? BIDS. % ' 
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2. Cocrounicaticms with ftpplicantHj(vestigatoirs^ . 

Sumnaii^tatenients relating the essential elonents of a review 
• groi5)'s* considerations and recoAm«xlaticMTS are prepared by the 
executive sfecrAtary of that particular review group. Eadi 
summary statanent follows a standard' format dn pr^enting a 
set of information poirtts descril^ingf the plication and 
explaining and justifying the review* group's recommendatipns 
r^arding the pertinen^ application and includes a priority ( 
spore whidi expresses, ' in numerical termsj the group's assess- 
n^t of the merits of^e applicaticwi.- Summary statements 
from IBGs are presented to the National Advisory Comcils for 
the final acticxis. In past years, investigatore requesting 
the results of their reviews have been provided with atetf acts - 
of the summary statements, usually by the BID program staff 
responsible for the particular applications and projects . 
involved. In more recait months, elenents of the Privac?y Act ♦ 
of 19"*^ ha^e led' to the release of -summary statanents to 
those irtvestigators who request them. . ' 



B. Issues • . 

Certain questions arSi problems have arisen in regard to both the 
referral process and c9mmunications about summary stateroehts^ 
whidi have led ta detailed ccxisideration by the GPRST:.- 

1^ Jteferral Process ' 

a. How can applicant- investigators 'have an input into the 

referral process, -in order to influence this process in 
\ a manner which would allow the appropriate dgci^ions to 

^made by NTH, but v*iich would take into consideratlcxi 
^•roncerns of investigators that their appliciatipns be 
reviewed by ^oups most appropriate aAd comjiejtent to do • 
' . ' > so? 

Jd.„ How can tiigy express their ccmcejns regarding certain / 
groups or irdividual advisors responsible for, such reviews, 
v*K>, it is felt, atre biased against the investigator or 
against the types of work or views suggested in the , 
. • application? . • . ^ 

* c. , How can investigators e55)ress their preferences that 
^ c5&rtain BIDs and corresponding National Advisory Councils 
considet their applications? ^ ^ " 

2. ConwiunicaticHis ^ * 

a. What .information should be released regafding the outcome 
of review groups' deliberations? Shbuld prior ity;scores / 
te included? ' 
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Td. How should this- ihformation'be cammunicatedli i..e., 

informal, oral communicationsoor by carplete, wifitten ^ 
critiC[ues? » • , / * 

.j:. To whpnr should the information be released, i.e., to a 
. principal investigator only,* or to institution officials,* 
or co-investigators? 

d. Vftien should /feuch information be released, i.e., iiwnediately 
, following, an IRG meeting, or following a Council meeting? 

e. Ho^ can iwestigators most expediently use v^iatever 
informatiorf^ released, in formulating and appealing 
elements ithey feel have "gone wrdig" in review of theix 
applications? * , • 



3. General Issues 



a. What elements of the review process should be madq subject 
to appeal? 

b; What recourse may be had to NIH off ids outside those 
which are currently responsible for actions or decisions 
which rightly are subjects for appeal? 

Approaches to sane of tdiese questions are covered in other portions 
,of thfe'GPRST R^rt. Aspects of these questions in\/6lving disputes, 
challenges, or other controversies regarding Nip peer, review processes 
are considered in this section. 

APPROACHES TO AN NIH GRANTS PEER REVIEW APPEAL^ SYSTEM 

A. Current Elements of an Pjppeals System 

It must be recognized that current NIH policies and procedures 
have been designed to handle certain aspects of the issues listed 
above With regard to BID ©r IRG assignment, for inslauice, ♦ 
investigators may, and. often do, suggest that applications be 
assigned to certain review groups or to certain BIDs. They may 
' challenge, the competence of the IRGe to which applications hav^ 
been assigned, or may question the ability of specific members r 
of .assign^ review groups to give j^ommendations without certain 
biases or prejudices against the applicanC institution, the 
investigator, or the ideas suggested in the application. Such 
communications are usually received by and handled within DRG, ' 
which is responsible for the assignment procedures, and sometimes 
•involve* discussion between DRG and the Bibs. 

Other procedures are ^vailable_to and are practiced by investi- 
gators whose applications are not funded due to the grant peer 
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review groups' recomroendation either for disapproval or for approval 
at priority scores which do not permit funding. After receiving' 
worl of, non-f iJsndinfi[ of ah application and of the reasons leading to 
such a decisicxir. the investigator may conmunicate further with the 
BID \i*iich is responsible for both the^ funding decisicJn, and, at 
leastr the initial communications to the applicant regarding that 
decision/ In return, the investigator, but not ''the applicant- 
institution, may learn more. detailed reasons for the reconmendation 
regarding th^ application; such information is based on the summary 
statement, and may be excerpted frari it; or the entire critique-text 
T may be sent to the investigator J While such ccmmunications are 
^conducted initially and primarily with the responsible BID? the 
investigator occasionally turns to the executive secretary of the 
; # IR3 responsible for the scientific review, to learn more regarding^ 
the scientific* raticxiale for the recommendation. 

. Furnished 'with these comments about the previous scientific reviews, 
the investigator is then free to submit a revised applicatJ.on, 
incorporating as many changes, as deemed desirable to meet the 
critical objections noted in regard to the original application, 
fiver the past few years, such revised applications have constituted 
approxynately ten per^gnt of the review workload. 

B. New Elements ^or a Formalized Grants Review Appeals Systerti . 

While designed to handle most disputes arising in connection wfth 
the peer review of grant applicati!bns, the procedures now in force 
have certain deficiencies. Basic to most of these is the fact 
that the mechanisms for challenge or appeal are informal, handled 
* on a person-to-t^rson, ^ h^c basis, 'without firmly established 
NIH policy. This, in turn, leads, to a corollary di^iculty in 
that the procedures are mot wi^Sely known, even though NJH has 
* , taken some, steps to provide accessibility of the revised application 
process to disappointed #applicants^ Moreover, the ^f act that , 
different kinds of disputes ajre l:\ajjflled mo?e or le^s entirely 
withij^i the same NIH conponent, or even the same office, otiginally 
responsible for the assignement or review procedure, has led - 
many investigatory to feel frustrated in their attempts to h 
seek' relief from outside that component or office. 

RECOWIEKDATION 1 ; , , 

' IHAT A FORMALIZED NIH -GRANTS PEER REVIEW APPEALS SYSTEM BE 

ESTABLISHED TO CORRECT Op ELIMINATE THE DEFICIENCIES DESCRIBED f 
ABOVE. . , * 

This system should have two phases to take into consideration 
those situatioj^s in which the assignment to a BID or an IRG may 
be djallenged, and those in which elements of the scient^ic 
•review may be challenged. In each case, the appeal must be made 
by the applicant-institutic«i (or individual) formally applying * 
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, for the grant, although t^e principal investigator may be. 
responsible for infohnal communications regarding scientific 

• aspects of the application. .While the appeal or dispute must be 
received first by the office responsible for previous acticxife, 
the new appeals processes, in both caises, require that other 
officials and offices soon enter into the consideration- and 
decision regard ipg the* arguments proposed by the applic^t and 
the investigator. • ' ' 

REOQMMENDATI(» 2 ; 

fSAT, IN ADDITION TO. IMPPQV^^NTS IN CURRENT PROCESSES FOR 
RECONSIDERATION OF ASSIGNMENTS OR REVIEWS; CfOTAIN ENTIRELy 
■ NEW MECHANISMS BE ESTftBLISte) FOR BOTO TYPES OP APPEAL TO . 
HANDLE SITUATIONS WHICHTSJIAND ACTICW OR DECISION OUTSiDE ' 
ORIGINAL NIH COMPOJEMIS AND AT A HIOiER LEVEL. 

I 

\..T^^, CENTRAL TO TfflSE NEW MECHANISMS BE THE ESTABLISHME^ OF 

. TOE POsiTicw OF ombudsm;^ appointed by the director, 




. ^ Many of the responsibilities .suggested below for receipt,. ^ 
processing, cbordinatibn,, and documentation of decisions regarding 
appeals of disputed, applications, should be assigned to the 
"Omtwdsman." . To be effective, such a f>erson'mustype highly 
experienced with the grants peer review system Jtn^t administra- 
tively independent of DRG.and pf the BIDs that' actively operate 
the system. " . / * , 

While it is felj: that it might' be most.,expedi/nt .to+iandle 
appeals entirely within DRG, at least for ce/tain types of 
appeals, it is recomnended that resolut^iqn involving DRG and 
applicants ^ould require a detadied thir<3 party. Hie Ombudsman 
; should be located in the Office of Extramural Research and 
Training' (OERT) , Nlfi, and should have the. responsibility and 
authority to coordinate the processing <^ grant peer review 
appeals. at higher levels. 

RECOWIENDATIQN 2b : 

...IHAT, TO PRD^DE .THE NEEDED HIGH LEVELS OF REVIEW RELATED -TO 
APPEALS, THERE BE ESTABLISHED: / ' ' 

,-a. A GRA NT^ PEER REV;iEW APPEALS BOARD" (GPRAB), A PERMANQJT ' 

OOmXTreE, THE MEMBERS OF WHI^H WOULD ACT IN MATTERS 
" CONCERNING ASSIGNMENT OF GR^ APPLICATIONS AND THOSE 
INVOLVING RECONSIDERATION QF SCIENTIFIC REVIEW OF 
. APPLICATIONS' (Table 1). * 

It should be chaired. by iLe Ombudsman, , should" include two 
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- - the same review cycle,^ If an assignment is 'appealed 
• aftet Review, the applicant must explain and justify . 
why tb6 appeal was hot entered earlier. 

2,i. After receipt of the sommarized'.conroents pertaining to^ 
. evaluation of tl^e, application; an applic^t-institution 
may appeal the-ev^yatiqn of the application if there 'is 
^ evidence that i^lvalid judgemehts, inadequate review, or 
factual err admiriistrative errors occurred in^the \ 
scientific review^ Such ^^peals may be made^n any 
.situation wfi^re the application does not re^lt in^a 
» • 1 • . gr^€ award;- National Advisory Council or. Bciard* program 
or policy recommendations may not be challenged, however. 

\ * It.is^elt that, to -a ccrisiderable degr^, the /proposed mechanis^T 
T describea below, for diallenging or appealing assignments or 

reviews will accommodate situa|^ions covering appeals and disputes, 
bef9re and after IRS re\^xew, and for all unfunded applKations, 
'whether favorably or unfavorably ,reconin»ded. It is also recog- 
nized , that , instructions for applicants must make clear ^at any 
appeal may result in delays ^uch'that the application may be 
held^untU the succeecjing Council round.'* Upon initiation of any * 
. ^ aK)eal action, the pertinent application would be withdrawn from • 
any further, review or processing action pending resolution of 
that appeal. A similar delay would apply to, any amend^ applic^- 
^ 't ion, 'resubmitted while appeal of the original application is being 
.considered. . Further, appeal #'any portic»i of a multi-project 
application e.g., •program project or center grant, would require 
that the entire application b^ considered undec^ppeal. 

^tEX3RNISMS AND PROCEDURES FQR APPeKlS CONCERMIWG ASSIGNM^ 'OF 
GRANT APP LICATIONS- 7~' ' ' '' — ; ^ 

RgCOMMENDATIQN 4 : ■ ., '' ' ' 

* • 

• . 'THAT MECHANISMS AM? PROCEDURES BE ESTABLISHED FOlR APPEALS* - ' 
CONCERNINS ASSIOJMENT OF GRANT APPLICATIONS. 

If not satisfied, upon notification Of an assignro^t to an IRG pr 
.BID, the applicant-institution 'must 'proceed as outlined below , 
(with number and letter ,teferehces to the corresponding lines • 
and columns in Figure ij: ' - * . 

An application (Line lA) submitted to the E»G ReferrsQ Office (IB);, 
^ . is assigned to an appropriate IRG and BID (2B). The applicartt-^, 
institution or its scierifeif it representative, the PI, receives 
notification of and considers the assignment (3A). in light of 
^ higher perceptions of the .review process. - * 



the same review cycles If an assignment is 'appealed 
aftet feyiew, the applicant must explain and justify 
why ttie appeal was hot entered earlier. 



2.1. After receijpt of the sammarized\coninents pertaining to 
. evaluation of the, application; an applic^t-institution 
may appeal the-evgj(ijratiqn of the applica^fion if there is 
evidence that invalid judgem^ts, inadequate revi^, or 
. ' factual qr admirlistrative errors occurred irVthe \ 
scientific review^ Such ^^peals may be made^n any 
.situation wHere the application does not result in^a 
♦ • 1 • , gr^n^ award;- National Advisory Council or .B6ar4* program 
or policy recommendations may not be challenged, however. 

\ * It.is^elt that, to a ccmiderable degr^ee, the /proposed mechanisms, 
^ described below, for diallenging or af^aling assignments or 

reviews will accomnodate situal^ions. covering appeals and disputes, 
bef9re and after IRS review, and for all unfunded applKations, 
'v*iether favorably or unfavorably ,reconin^ed. It is also recog- 
nized .that , instructions for applicants must make clear ^at any 
appeal may result ip delays ^uch^that the application may be 
held^until the succeec^ing Council round. ^ Upon initiation of any ♦ 
. ^ appeal action, the pertinent application would t>e wi'thdrawh from • 
any further, review or ^processing action pending resolution of 
that appeal. A similar delay would apply to .any am^^d applicfi- 
' 'tion,' resubmitted while at^a^of the original application is being 
considered. . Further, appeal Jl any portion of a multi-project 
application e.g. program project or center grant, would require 
that the entire application be considered undep^ppeal. 

flEiqmNJSMS AND PROCEDURES PQR APPeKlS CONCERMING ASSIGNM^T OT 
gtfVNT APPL ICATIONS- T" ^ ' ' — ; ^ 

RECOMMENDATION 4 ; ■ " • - ' ^ 

• . *THAT MEX^iANISMS flNEJ PROCEDURES BE ESTABLISHED FOl^ APPEALS* - ' 
CONCERNING ASSIGNMENT OF GRANT APPLICATIOJS. 

w • 

If not satisfied, upcxi notification of an assignro^t to an IRG pr 
.BID, the applicapit- institution 'must proceed as outlined below 
(with number and letter .references to the corresponding lines 
and columns in Figure 1): ' - ' , 

An application (Line lA) submitted to the YSIG Referred Office (IB);, 
is assigned to an apprc^riate IRG and BID (2B). T^e applicartt-^, 
institution or its scieri^ifit representative, the PI., receives 
notification of and. con3iders the assignment (3A). in light of 
' hi'^/her perceptions of the ^c^view process. - ' 



••'If dUsatisfied, the applicant may request {4A) that the appli- 
cation be assigned to a' different BID or IPG, o? that s^)ecif ic 
reviewer (s) be excluded from the review, reasons' for vrtiidj ntust 
be detailed." Upon receipt of such a request (4B), the Iteferral 
Office decides (58) oh its validity. This appeal is coimunicatejJ 
to the Associate Dfreptor for Scientific Review, DBG (6B), "for 
>his review and conc4r$iie€<'i<7B) .. This is the-,first step at' which 
the assignment is agiaidered at a level above that where the 
decision was made oli^inally; U]^ receiving a conmunication 
about the DPG decision, Ihe applicant considers tj^ei^ction (8A) , 
and decides either to accept the decision (9A) or to subrtit a 
further appeal (lOA). • 

If the applicant wishes t^ continue the appeal, it (including 
justification^ is sent to' the NIH Otrijudsman (IOC). The Ombudsman 
will form a group, (IIC) consisting of Himself /her self plus two 
members of , the Grants Peer Review Appeals Board (Table 2) vrfio are 
from Nip con?»nents not involved in the = appeal (12C),. Diis group 
constitutes the nekt -higher level of "e^jpeal regarding assignments, 
These three individuals will consult (130 Jwith the Executive 
.Secretary of the involved IBG (14C), the staff of the BID 
involved (15C), artd the Referral Off ice (16C)- but then, will 
reach an iildependent decisions^ the ^ropriate assignmefit (17C). 

V If this decision is to lifiiold' the original assignm^t (18C), this' 
is cottimirticated to the applicant institution. There i-s -no further 
recourse (22A) . If, however, it is decid^ to change, the assign- 
ment (19C) this decision is communicated to the applicant and to 
the Referral Office, DRG t20B) which will process the application 
as required" by that decision J 21B) . The applicant will hav^ no 
further recourse in regard to this revised assignment (22A). 
> . ' ' 

PROPOsit) -MECHANISMS AND PROCEDURES FOR APPEALS OOWCERNING SCIEOTIFIC 

REVIEW 6^ GRANT APPLICATIONS '■ ' 

RfXJeroiENDATtOW 5 ! ■ • , • ' 

THAT MECHANISMS AND PROCEDURES ,BE ESTABLISHED FOR APPEALS 
OONCESNING SCIENTIFIC REVIEW OF GRANT APPLICATIONS. , ^ 

An appeal ccaicerning the scientific review t>f a gtant application 
will begin only after the PI receives the suirmaf ized conments of 
the review regarding the application as presented in the suninary 
statement. Upon consideration o( these cannments the BI has three 
optioiB: (1) to. accept the sunmarized comments without further 
' action; (2) to revise the application in response to the 

• summarized comments sni proceed" as described below under A., 
"Resubmission for Review"; or (3) to invoke the appeals gpxess 

• as described below Jhder B., "Appeal of the 'Scientific Review." 

* As illustrated in Figure 2,. appeal of the scientific review of 
a grant application starts with issuance by tfte BID of the • 
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:suninarized coninerits of the scientific review (IB). IWte PI * . 
cogpiders these review^ comments. (2;^) and may decide, as one - 
: option, to takfe nb further action (3A) or^ as the other: option, / 

to revise anS resubmit the application (4A). ^' / 

* , * • 

i* Resubmission for Review • , i * * 

* . ' - ' 1 .'..•'/ ' • 
This form of appeal has always been available, to tte a^licant 
and^rren^y is used to a 'Significant extemt, , 

) ' * * . ' ' ' 

• If the plication is to be resubnitted, it should be revised in 
response to the summarized ccninents by the addition of new 
information , or other ain^xlments,- Whatever roodificatiens are made 
to the plication roust be indicated In a s^r ate 'covering letter 
dcconpanying the resubniaisiort, ^'itiis will help reviewers in their 
reevaluation of the proposal and will also help present sudi ~ 
material fi;pm bejLng overlooked,, F^iilure to point out these 
modifications could result in the application being returned with- 
out further review.. Suggestions regarding IHG referral "or use of 
specific reviewers may accorapan^ the re^j^itted application. * ^ 

All this material $pd informSttion is submitted to the-Refefr;^ 
Office, DFG (4&) which will consider- the? documents and suggesticxis' 
and assign toe revised application to an appropriate IRG (SB). • 
The latter vail review the application during the next appropriate 
review cycle (6B), and its recommendatiorts, wiil be considered 
by the appropriate BID Council/Board (7B). If the applicant- 
institution, meanv*iile, ctjrisiders either of these two assignments 
in^ippropriate, a new assignment may be requested through ^e '* 
mechanism destfibed above in Section IV. ^ ' 

Appeal of the Scientific Review ^ 

If the applicant-institution belie^^es therg^were errors in the 
original review, '^an appeal of the evaluation may be made by 
Requesting reconsider atic»i of the review, with rebuttal of the 
comments (8A) . Such a request should be sent to the Blli which 
issued, the results of the previous review (8B) apd should give 
the reasons for the request for reconsideratic^, taut subnit no - 
modifications to the plication , ihe applicant should subnTF'only 
such iijformation as is 2?)propriate to support this request for re- 
review, and must initiate the request withiinf 90 days after the 
mailing date of the slmnarized coramentSj, and. In any event, not 
later thkn 120 days after the Councii/Board meeting. 

— # . ^ ' 

The BID staff will consider the validity of Jthis request for * 
reconsideration (9B) fuid, in this pro6ess, will consult with the 
aipropriate staff of Drg and the IRG responsible, for the initial 
scientific review (lOB). The BID may accqpt the validity df the. 
request (12B) and maj^ therefor^ return the materials to the IRS 
for rereview in the ne^xt cycle (13B) or to. the BID Council/Board 
for its ccmsideration (15B).f 
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If however- the Bia staff considers the request- for reconsideration 
invalid {UB)v^this decision.will be conmunicated to ttle institution 
{12A)', whitfR^may decide either to accept (14A)-or appeal this 
decisibn (16A). Such an appeal should "be' dlrectd to the NIH 

• .Onbudsinan,(16C), giving reasons fdr the request, but again 

• submitting no modifications to the application , ^s mentioned ^ 
above. >1he applicant should suh^iit only such information as is. 
appropriate to support this request for, re^r^view, and imist ^ ; , 
initiate this request within 20 wcarking days after the mailing 
date of the BID communication denying the request for^ 
reconsider at it)n (IIB). , • 

* ' ' * ' 

■rfe bntoudsman will cbnsult with. two merbers of the C3>RAB (17C), 
from NIH corponents uninvolved in the appeal (18C). This, ad .hoc ,^ 

' qtoap may decide to reject the request as one not falling withm 
the jurisdiction of the peer'^eview appeals system (19C), oi: to • 
consider the request further ^through form^ion of a Grants 
Review Appeals Panel (20C) (see also Table. 1). The Panel may 
'decide (21C) to reject the instityit ion's appeal ■ (22C) • Neither 
the ad^hoc group rejection (19C1 not the GPRAP's rejection (22C) 

' may ^ale4 further by the applicant (21/22A). The Paoel ' 
may decide, however, that the applicatifl(n deserves further 
consideration by an IPG (23a) or by an Sq2j)ropr iate Coungil/Board 
: .(24B) an d iBay " g ^act. In dckng so, the Panel should" have ^ 

authority to de^e to what Extent the application was adversely • 
affected in the review and to direct the IR3 or Council /Board to 
reconsider previous actions. The NIH appeal process ends with, 
the decision of this Panel, which is reported to the applicant, 

• the BID, and to DRG. , ) • . . ) ■ 

cx)wcmsi6w ^ . V * . 

The Grants Peer Review §tudy Team believes th&t these recoitroended 

• mechanisms involve both improvements on curr-ent procedures and 

■ suggest ibhs for view ones. These principles for an Appeals System 
will futnish the basis for ne* and improved NIH policy and. 
' procedures to .handle disputes and should, in turn, help greatly 
towards better understanding by and equity^frsf ^members of the 
^ • broad biomedical Scientific cormunity. - " 

y ; ' ■ 
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. TABLE 1 . ^ 
GIANTS PEER REVIEW 'AP PEAL GROUfe 



Grants Peer Review Appeals Board - GPRAB 

.AHwinted by Associate Director for Extrainur2LL Ites^arch and Training' (ADEPT), NIH 
" MelDbers ; 1 - Sert - OirbudSman - diair 

1 , ; 2 - (X> - permanent ' • . ' 

3 - ECEA (BHDs) - rotate/alterfiate ' .' . • . 

' . ■ - ' ( 

Executive Siecretary - appointed ty ADERT . 

- • _ - ^ . <s 



Grants Peer Review Appeals Panel - GPRAP 

Selected by Ombudsman A * • ^ ^ . ^ 

Mentoers ; 1 - Ombudsman ^ chair • ^ ' , ' 4 

2 - GPRAB U CD + 1 BIDr from uninvolved BIDs) * 

^ 2 - NIH- ex^Iamural staff (ad hoc) >^ ' 

1 + Additional scientif ic/acSnTnistative/public consultants, 

^ &s required ^ . - 

Executive Secretary - selected by. Ombudsman • ^ , 
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FIGURE V 

''actions' TO -RECONSID^ ASSIGNMENT OF (3RANJ APPLICATIONS 

•TO IRG OR BID 
. ' 
• ■ (A) • • (B).. 

"APPLICANT. 



(c) 

OERT. 



1 Sutjmit application 

2 ■ ' - ' . 
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• ; • ■. I'- ;•. 
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22 • No recourse <^ 
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■ ■\ 
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>I^KeferraV.Office • , 
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^ • '. . - 4 - 
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AAMC Association Of American Medical Colleges 

ADAMHA - Alcohol, Drug Abuse, and Meh^ Health Administration « 

jffiERT - Associate Directpr for Exttamural Research and Training 

AC6R - Associate. Director for Scientific Review ^ 

AIBS - American Institute'of Biological Sciences 

» ♦ r 1 

ARA - Awaiting receipt of application (a file) 
, A^C Availability Request Clearance 

- Assistant Secretary for Health (in DHEW) 

\ ^ 

BHM - Bureau of Health Manpower » ^ 

BID - Bureaus/ Institutes, Divisions 

BMC -^Bu^iness Management Consul1;ant 

CCC - Comprehensive Cancer Centos 

Z.yX. - Code of Federal Regulations 

CMO - Ccntniittee Management Office^r) ^ 

GSRS - Cooperative State Research Service (USDA) 

CV - Curriculum vitae 

DBER - Division of Biomedical and Environme|gS^ Research (ERDA) 
DCG^ - Division of Contracts and Grants 
D.D.C. - District Court of the District of Colvinbia(^ 
DHEW - Department of Jea^th," Education, and Welfare 
DMP^ - Division of Management Policy , 
DBG - Division of Researdi Grants 
DRR - D^[s^r*of Research Resources 
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BCEA • 
eBDA 
E^BAC 
FACA 
POIA 

F. Svpp. 

' GMAC 
GPRAB 
GPRAP 
GP^ 
HEW 

HEW-474 
HEW-532 



^ Executive Catmittee for. Extramural Acfclvitlles 
- Energy Research and Deyeloproent A(3mioistration' 



- ^ecutive Secretaries' Review Activikes Ccmittee 

- Federal Advisory Coirpitt*e Act 

- Freedom Of Informatioi Act / . 





Federal Regist'el: . 
Federal Ctpfjlot^ent 
Pi'scal^Year - / ' 
Crants' Managonent Ad\risory' Coram it tee . 
Grants Peer- Review ^^jpeals Board* "^^^ 
Grants Peer ^ Review Advisory Panel* 
Grants Peer Review ^tipyV^ 
Departmoit of Health, 



■J 



tior\ , ^ and * Wei far e 



- Confidential Stat^nt of Bnployftient and inane ial Int^ests 

- Request ^9^ Approval of* Nominees jfor Public Advisory 
Coram ittees ^ 



HEW NJH OD 0300 - Designation for the off irtal NI|i files conGerning grants, 
fellowships, and ce^t^in other types of awards 

H.R.QOO • House of Represen^tives Bill 

H. Ifep. -yHouse Report 

IMPAC 



IRS 



Inforinatioa for Manag^went, Planning, Analysis,] and Coor<3inatiofli; 
the computers-based record-jceeping system-, of 

- Initial Review Group ' it 



JEDLAR - Medical Literature"" Analysis and Itetrieval Systfem. (of National 
Library of Medicine) 

MBDLINE - MEDLARS on-line service 

MRC - Medical Research Council 



♦Projected 
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NICHD 

NIOR 

NIEHS 

i 



" • NIH 



NIMH 
NINCDS 

* 

NLM 

-NSF. ' J 

OCR 
^OD^ - 

.OEBT 

* 

OGC ' 

OLS 

ONR 



- Natiaial Mvisoty Council/Board 

- <National Acadany of Sciences 

- National Aeronautics and Space .Administration ^ * 

- National Cancer InsUtute , 

- Natiorua Eye Institute 

- National -Heart, Lung, and Blood Institute 
' - National Institute on Aging 

' - National Institute of Allergy and Infectious Diseases 

- National Institute of Arthritis, Metabolifift, and 
Digestive Diseases 

National Institute of Child Health and Human Development 
. -I ■ ^ ' . . 

- National Institute of Dental Research ^ 

*- National Institilte of Envir^Dnmental Health Sciences ^ 

- National Institute of General Medical Sciences 

- tJational Institutes of Health 

- National Institute of Mental Health 

* ' • ' ' , 

- National 'Institute of Nejjrological and Comnunicative ^ 
Disorders ctnd Stroke 

- National 'Libr'firy of Medicine ' ' . 

- National Science Foundation 

- Office of Collaborative Researdi 

- Office of the Director, NIH ^ ' - 

- Office of Exttamural^6earch. and Training^ 



- Office of General Counsel 

- Office of Life Sciences (NASA) 
-Otivz^ of Naval Research 
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Office of Program J?iamning and Evaluation 
Privacy Act, 

Public Health Service • 
Principal Investigator 
Public Law - ' * 

President's Science Advisory Conmittee • 
Project Site Visit 
Referral Branch 
Request for Application 
Request for Proposal 

Investigator-initiated .research grapt' application 
Statistics and Analysis. Br ancff 
Subconroittee onBusinessi Management Practices i 
Specialized Centers of Researdi. 
Senate Bill 

S4 Rep^ - Senate Report' ' ^ 

Special Initial Review Groups ' 

Scientific Review. Branch of the bivision'of Res.earch Grants 
Study Section 
Special Study Section- 
Onitet? States Code ^ 
United States 'Dq>art5nent\>f Agr^iculture.' 
Veterans Adminlstraticx^ 
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